princeton-cdh / bitklavier

The piano, digitally reinvented.

bitKlavier: the prepared digital piano

Developed by Dan Trueman, Michael Mulshine, Matt Wang, Davis Polito, Theo Trevisan, Katie Chou, Jeff Gordon, and Camy Streuly.

bitKlavier takes inspiration from John Cage's prepared piano, but instead of screws and erasers we place a reconfigurable collection of digital machines between the virtual strings of the digital piano. Learn more at the bitKlavier website.

bitKlavier is built with JUCE, the C++ audio programming framework (available under GPLv3 license).

Development on bitKlavier is sponsored by the Center for Digital Humanities @ Princeton University. See the CDH project page for more details.

License

This project is made available under the GPLv3 license

Installation

Visit the releases page to download the latest version. You also need to download a resource package with samples and galleries; this resource folder must be placed in your Documents folder.

Development setup and instructions

Branches in this git repository are named based on git flow branching conventions; master contains code for the current release, new feature development for the next release is in dev.

Download JUCE and Projucer for your platform, install the appropriate compiler and devlopment tools, and open the project in Projucer. Code is structured in the standard Model-View-Controller design pattern.

To compile bitKlavier as a VST plugin, you will also need Steinberg's VST-SDK.

To run your locally compiled version of bitKlavier, you will also need the resource folder (see Installation above).

bitklavier's Issues

nostalgic undertowPreparations question

i'm currently saving a pointer to the current nPrep so that when the undertow note is started it uses that preparation and not the current preparation when the undertow note begins, should the preparations have changed since the note was first played. it occurs to me that this could be unsafe in the rare circumstance where there is a nostalgic note playing while the user opens a new library, resetting all the preparations pointers, in which case perhaps nPrep should probably be actually copied to undertowPreparations. we should not forget to evaluate this, as we further development how preparations/presets/libraries are handled.... at the moment it's moot as we only have one preparation, but obviously this will change...

get vs direct preparation data access

wondering whether (as MM mentioned) we should be using these preparation->getVarName() calls or accessing the var names directly. since these get() functions are all const and noexcept, it is probably just as efficient as accessing directly, but just wondering...

Changing number of Layers

Need to deal with situations in which number of layers changes during playback. Not too tricky.

If we "remove" a layer (disable a Processor), then the associated Preparation data should be wiped/reset. This will take care of "adding" layers too (enabling a Processor).

Synchronic (non-threading) tests

Tests with many layers (i.e. 20+):

After pulses have begun, but still in cluster time range, adding and removing notes causes blips.
Adding notes to cluster twice can cause blips (bug).
The pulses themselves are clean, even with 100+ layers. (impressive)

Nostalgic SynchronicSyncMode quirkiness

Variable behavior. Try setting beatsToSkip to 5 and tapping notes (and Synchronic numPulses to 10 or so to hear the pulse).

I think this has something to do with the SynchronicProcessor phasor/beatLength not being updated before nostalgic keyPressed is received.

Nostalgic reverse playback (from past end of sample)

Working on it.

Dan ToDo

• basic documentation for beta-testers; google doc that some of them can edit
• test test test... both playing directly and embedding in Sibelius to reproduce Etudes and Mikroetudes
• create clean bitKlavier_examples, and NostalgicSynchronic galleries
• forum, initially for beta-testers; forum hosting sites?
• import/export individual preparations
• add global gains for Direct, Nostalgic, Synchronic to General Settings

---

• resizing doesn't work properly with Standalone on my laptop; works ok for Mike, and also on my iMac
• titlebar color!!! not addressable in plugin, but possibly in standalone; might have to put Options button functionality elsewhere
• splash screen for standalone

---

• envelopes for Synchronic, others?
• implement "relative to tuning" transposition in all preparations
• implement pedal down and up samples
• Plugin automation? going to be tricky, since the available params are so dynamic, depending on the Gallery/Piano configuration. this may be best to wait for a later version
• transposition bug; 037 in Direct, for example, lots of dropped notes. related to minor-3rd separation of sample library?
• tuning names are not being saved to XML
• Marbles behavior is not quite right; notes are not sustained long enough in the pulses.
• tempo note-length mode not getting set properly
• tempo reset button not working
• figure out what is going on with Nostalgic sync mode when Synchronic is in anything other than First Note-On
• single click on Nostalgic trans slider doesn't change value unless close to slider. helps with double=click
• Synchronic transpositions; add new with value near click, and background alpha with multiple transpose
• General Settings: only really need tuning reference (440) and tempo multiplier
• click-away issues in Preps; want to be able to click outside and have it go away. also click INSIDE and have Texteditors lose focus
• default tempo on gallery load
• "clear mods" button in Mods UI
• UI for Mods
• hide noteOff Loudness
• dropShadow for construction site?
• transposition sub slider stuff in Synchronic
• keymap keyboard; select multiples by drag... also for playback in the bottom keyboard
• bKeditableComboBox crash; click on arrow, then menu
• Synchronic gain slider not doing anything
• KeyMapVC
• sustain pedal not working properly with preps; they start with keyUps, regardless of what pedal is doing
• work up rough UIs for all preparations
• test json loading and old presets
• UI mockups of preparation windows, modifications windows, and related
• work on transposition issue when preparation changes while key is down
• Tempo Preparation; create, extract from Synchronic
• Plugin host saving/loading Presets
• reset keymap implement storage/loading, but not resetkeymap mod, which is just perverse
• reset keymap, within preparations; specific keys copy static to active and reset internal params like adaptiveTempoMult and adaptiveTuningFundamental
• implement modifications for Absolute tuning
• Heavy nostalgic clicks
• accent/gain vals > 1 in Synchronic not working
• mockup UI for Modifications, with new single-key assignments
• absolute tuning; offset for any particular note(s) (64:-13.7, etc...); add UI for that
• custom tuning with arbitrary number of offsets; rotating tuning. main question: which octave does the fundamental apply to? the lowest?
• implement adaptive tempo
• global tempoMultiplier
• global tuning fundamental
• is synchronic tracking velocity to set loudness of pulses? need to implement
• same question for nostalgic; seems not... fixed.
• modify Max mockups (wrapping preps/pianos into menus), refine
• fix adaptive tuning
• implement custom tuning
• implement direct toggle for resonance and hammer
• repeated notes => clicks with resonance/hammer; check into
• edit hammer and resonance samples to make sure they don't have pre-sound silence.
• check to see if Direct layers cause sounding notes to cut-off, if they use the same sample? need to check. similar to issue with Synchronic.
• check to see if Transposition is being handled properly; we want to choose the closest sample after taking into account transposition

channels

wondering if we have fully implemented "channel." this may not matter, but it also might when utilized as a plugin, with multiples. should be easy to do, but we should confirm as we go forward...

Click when repeating notes with Hammer/Resonance samples On

When the hammer/resonance samples are allowed to play, fast repeated notes (especially fast repeated chords) cause clicks... Need to be smarter about retriggering hammer/release samples, especially when they are retriggered before the end of the previous.

Mike's TODO (beta)

• Scrolling, zoom
• Undo/redo
• Copy/paste
• Patch cables
• Add/delete individual preparations from dropdown menus. Manage deletion of unused Items.

noteIndex offset?

why do we sometimes have:

int noteIndex = m.getNoteNumber()-9;

with the -9?

i'm not finding it necessary for Nostalgic, and we should find a way to make it consistent...

UI First effort

Storing current global preset state

Useful resources/potential techniques:

1. ValueTree (definitely useful, easily converts to XML). Will test.

• https://www.juce.com/doc/classValueTree

2. SharedResourcePointer + SharedResourceManager

• https://www.juce.com/doc/classSharedResourcePointer
• https://www.juce.com/doc/classSharedResourcePointer

3. [NO, DON'T USE] Singleton (https://www.juce.com/doc/juce__Singleton_8h)

• https://forum.juce.com/t/lazy-initialization-for-juce-singletons/6761

4. PropertiesSet and PropertiesFile (https://www.juce.com/doc/classPropertySet)

GeneralSettings ToDo

• tuning fundamental frequency
• invert sustain (should be relatively straightforward within Synthesiser class)

Direct ToDo

• overlay

• integrate transposition with Tuning

• gain

• merged with Main, so Direct becomes Main

• rename Direct as Main?

• resonance and hammer? have a toggle on each layer whether to trigger them or not, by default of this toggle on for layer 1 and off for all the other layers.

• will layers cause sounding notes to cut-off, if they use the same sample? need to check. similar to issue with Synchronic.

beatsToSkip and nostalgicSync mode

beatsToSkip (Nostalgic) scales numSamplesBeat (acquired from SynchronicProcessor) to determine the duration of the reverse note to play on note off. But each synchronic beat may be a different length. Shouldn't we sum the number of samples in the next [beatsToSkip] notes in order to properly set the playback duration?

Keymap should be passed as pointer to Processor

Multiple preparations assignable to Keymap/Tuning pair.

Minimal clicking when playing TONS of notes with pedal down

If I keep the pedal down and play literally hundreds of notes within a span of 5-10 seconds, there are a few clicks here and there... lol. This may not be an issue, but it might also be worth trying to get rid of early on.

It may just be that we haven't thought of any particular cases in which having the pedal down may change the way we trigger samples.

NoteOffSync

Still need to implement synchronic note off functionality.

Layer mute/mixer.

Muting layers would be awesome (instead of removing them or changing an obscure parameter to hear other layers). I guess this would be built in to a layer mixer unit if/when we get there.

Tuning ToDo

• Implement TuningPreparation class, pass pointer to S/N processors

• Implement Adaptive Tuning functionality

• adjust and complete GUI to fully test, with all Tuning params

• fix adaptive

• implement custom scale

UI Todo

• Side bar (Displays preparations, modifications, keymaps, etc.; left sidebar => level meter?)
• Click and drag to PrepMaps

Reconfiguration

Layer Configuration:

[ layerType ] - [ layerNum ] - [ keymap ] - [ preparation ] - [ tuning ]

:D

Json load test bugs

• P: the Tuning1 key mods get setup and work properly on load, but they do not show in the mod setup field, nor do the export properly when then saving to xml

• P: Direct1 configures things incorrectly; should load two keymaps, one with everything but 60/62/64, the other with just 60/62/64, and then map those to two different Direct preps with different gain vals

• P: Preset Keymap examples are working, except that PresetKeymap1, the lower C should be muted, so this is another example of the Direct not being setup properly

• P: Adaptive Tempo presets aren't working; looks like the modes aren't implemented, or reading.
  a. just need to add keymap; not sure i'm going to bother adding this to the json read, as it's a bit messy

• P: Tuning2: the "scale" should be 6
  a. but, even when set to 6, this preset is not working properly

• need to fix cluster min/max so that inverted works, when min > max

• timing of reverse notes in Sync mode is off again; they peak too soon. i thought i had fixed this!

• P: Layer examples in v2 seem to work, Synchronic lengths need to be adjusted...

• Nostalgic4: reverse notes should start when key is pressed! not when released.

• Synchronic9: there is a bug with the first beat-length multiplier; try setting transp vals of 4 2 0, and when the keys are release, the first beat is super short

• P: Synchronic3: BeatsToSkip should be -1

• P: need to convert synchronic length mults from multiples of 50::ms to multiples of the beat length as set by tempo
  a. so, Synchronic 1/2, which are at 120bpm, should be 0.1
  b. LM_new = LM_orig * 50 * tempo/60000

P: => json preset loading issues

Modifications Design

fixed modifications:

• original preparation, never changed
• initially, original is cloned to active preparation
• processor points to active preparation always
• user creates modification, which creates clone of active, user modifies clone with new target values
• when modified is invoked via keymap, active becomes the clone (processor points to clone)
• when switching pianos, active preparation is always cloned from original, so starting point will always be the same

thinking forward to iterative modifications:

• modifying function is called every time the modification is invoked, and active value is calculated based on previous active value

New Keymap/(Piano) concept

Keymap construction

Keymaps are collections of one or more notes. KeymapLayerId becomes KeymapLayerIds (a sort of keymap construction area) where Keys and Keymaps are combined to form a larger set of keys associated with a Piano.

So, say there are several keymaps: K1,...,Kx.

Originally we could just do K1 or K2, meaning that our Piano processes keys in either Keymap K1 or Keymap K2. With the new method, we might be able to do [K1 K5 C4 62] which would add all the keys in K1, all the keys in K5, midi notes 60 (C4) and 62 to the Piano's larger Keymap.

One step further

This also has larger implications for how we might go about assigning PreparationMappings (P1,...,Px) to individual keys or Keymaps. For example, what is now called KeymapLayerIds (but maybe should be called Key/KeymapConstructor), could look like:

K1 K5 60 62:P1 K3:P2

This would result in a Piano that contains all the keys in K1, K3, K5, 60, and 62. Plus, 62 would cause a Parameter change P1 and any key in K3 would cause a Parameter change P2.

Another step further...

If we took it one further and eliminated the notion of the Piano (as we talked about), we could have keymaps like:

K1:[S1 N2] K5:D1 62:P1 K3:P2

And that would basically be a Preset. When 62 or any key in K3 is hit, we have a new configuration of Keymaps/Preparations/Mappings, thus a new preset.

The problem, of course, is how to represent this "Another step further..." in the UI. I bet there is a way. In the end, I think that at least having the KeymapConstruction area (keeping the notion of Piano intact) would be cool and useful. Thoughts?

hammer samples

are not edited carefully enough, resulting in a slow response; need to edit each one by hand. funfunfun!

Modularize components (MVC)

https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller

i.e. SynchronicViewElements <=> SynchronicViewController <=> SynchronicPreparation/Processor

SynchronicView: (View)

Knows next to nothing about state of model - just presents SynchronicPreparation parameter data in UI, enables editing.

SynchronicViewController: (Controller)

Parses input from the View, routes it to the Model.

SynchronicPreparation/Processor: (Model)

Keyboard input processor and parameter change processing happens here. Notes are triggered.

Overlap or NoOverlap type

...to prevent NoteOns from turning previously allocated notes off before allocating their own.

Click at end of reverse playback

Probably reading past end of sample?

GUI Organization

Useful view objects to implement:

• Label+TextField
• Dynamic/Resizable Row/Column<Label+TextField>
• Row/Column abstractions which set the frames of multiple UI objects that are potentially different.

Unit Testing

Check out:

1. Jenkins - https://jenkins.io/solutions/c/
2. David Luet
3. Check out JUCE unit-testing community

Performance glitches introduced

Performance glitches introduced when iterating through multiple layers, particularly on noteOffs. Nostalgic keyOff/playNote can be improved, but it may make sense to think about threading the xProcessor stuff soon.

Abstract super classes for Processor, Preparation, and ViewController

Will make development and debugging easier.

• Processor abstraction
• Preparation abstraction
• ViewController abstraction
• BKLabel+BKTextField abstraction

Synchronic ToDo

• note off sync

• play only when notes off / play when notes held toggle

Pointer to sProcessor array should be passed to all nProcessors for syncTarget

Preset categories/questions

there are a lot of questions regarding how to structure presets. the current version of bK stores the current settings for all preparations, including keymaps, but does NOT store General Settings. the rationale for this is that the GS are things you might want to change for a particular context (tuning to A442, rehearsing at a slower tempo, etc...) without having to go in and edit all the presets individually. new things like we are considering (solo/mute, a mixer for preparations, etc...) seem to me to be similar.

however, we are considering making sub-presets, so you can export/import just a particular preparation setting, or particular keymap (it's easy to imagine composers building libraries of these individual components), so it's fair to ask whether we should do something similar for General Settings and perhaps the Mixer. it also begs the question whether something like a Snapshot, that captures EVERYTHING about the current state and can be recalled, might be useful.

and how this would all relate to Plugin preset structures is unclear to me.... i could imagine a Plugin preset really making more sense as a kind of Snapshot as i just described, whereas as presets saved to files and shared would be more like how it works currently, with the addition of sub-presets.

food for thought....

Preparation-independent gain

In addition to global gain controls for Synchronic/Nostalgic/Direct/Resonance/Hammer, do we want preparation-independent gain controls?

Direct would contain DirectGain/ResonanceGain/HammerGain, for example.

To Do

• finish dummy GUI with all params
• Keymap
• General/Main settings
• Direct
• Tempo
• Tuning
• Presets

SynchronicPreparation type

Need to make a SynchronicPreparation class which acts as a data wrapper for the parameters of a synchronic preparation/layer.

This should interface smoothly with the SynchronicProcessor class. Each Processor class is tethered to one Preparation class. When parameter values in the Preparation change, the Processor should reflect those changes. (Similar to Sound/Voice dichotomy in current synthesiser implementation).

Synchronic fixes/adjustments

1. I think it's important at this stage to keep the parameter interface definitions the same as the current release (with the exception of the Sync Mode), and i'd like to always talk in ms for now. i am changing the "clusterThresh" to be in straight ms, as opposed to a multiplier of the pulse length as it is now. we can add the multiplier approach later, but for now to avoid confusion, I need the original definition

2. removed (!inPulses) from conditional for keyReleased; when in LastNoteOffSync, we should be able to interrupt a cluster
   => more to the story; see 4

3. it sounds like repeated notes get added multiple times when in a cluster, so they get louder and louder. changing to "cluster.addIfNotAlreadyThere(noteNumber);"
   =>actually, a bit more complicated. see 7 below.

4. when in LastNoteOffSync, the pulses should not begin until ALL the keys are released; we need to keep track of which keys are down. it looks like the "on" array was meant for that, and is unused otherwise, so i've implemented this using it, and renamed it "keysDepressed"

5. changed rampOff time to 30ms, to match current release; makes a clear difference to the sound. at some point we may want the user to be able to set these individually within the preparation

6. added "if(LastNoteOffSync) shouldPlay = false;" to keyPressed() -- sometimes the pulses would continue when a note was pressed.

7. so it turns out that bK has an effective max cluster size of... 8 and if more than that are added to a cluster, then the oldest ones are replaced. without this, rapid passages can result in a huge build-up. 8 is stupid arbitrary (should be 10 at least), but i'm going to put that in there so we can reproduce the sound of existing preparations, but we may want to make it settable in the Synchronic preparations. doh!
   => I've added a "clusterCap" variable to the Preparation, set to 8 for now; comments in the code as to how clusterCap and clusterMax differ. we may want to expose clusterCap to the user eventually, as long as we can do it without sowing confusion; it may also be just one of those things that remains hidden as part of the design of the instrument.

8. ok! i've gotten Etude #1 to sound the way it should, playing as a plugin within Sibelius! there is only one caveat: there is a little strangeness with playback when velocity is 127 (it seems to poke out really loud all of a sudden). i wonder if that is possibly a consequence of the reduced sample setup we are currently using now though, so i'm not sure i want to spend more time on it until we try all the samples.

9. i don't think we are doing the tuning of resonance properly yet! trying this with Etude 2.
   => fixed.

10. rename "beatsToSkip" to "offset"?

Piano Feel tuning.

Just fooling around with a full keyboard, I've begun to think about ways we should consider "tuning" the feel of the piano (and perhaps make the touch/feel user-customizable). I think this stuff will be pretty important if we want to make something that can really function as a piano.

1. Sample delay: We can vary the start time of the sample between 0 and 100+ ms to change the perceived Latency/Brightness.

2. Velocity layer cutoffs: We can change the Velocity layer cutoffs to change the perceived Brightness and overall Feel.

3. Gain of Low vs. High: Low notes seem to poke out more, and I can't really get a high note to stick out when i really punch at it... Maybe some kind of exponential gain curve across the keys (could be dynamically changed) to variously attenuate/amplify different ranges of the keyboard.

4. We intended to do this, but make sure the Hammer/Release samples happen instantaneously and consistently across keybed.

Synchronic FirstNoteSync issue

Uncertain of proper FirstNoteSync functionality.

Dan, when you are in FirstNoteSync mode, and it's been longer than the clusterThreshold since you played the first note, what behavior is expected?

i.e. Assume clusterThreshold == 500ms. If I play a C60 and then 200ms later play a E64, the synchronic notes will not be started until 700ms after the start of the first note (since the E64 resets the clusterTimer). What behavior is expected in this case? Right now, this results in a little 200ms glitch when otherwise you'd expect to hear pulses starting as soon as 500ms after the first note.
the sounds at once.

Synchronic - Relationship between Cluster Threshold and Tempo in FirstNoteSync mode

What happens if the period of the tempo is smaller than the clusterThreshold in FirstNoteSync modes?

i.e. Tempo = 150 (period == 400 ms), clusterThreshold = 500ms, beatsToSkip = 0 . This test case within current functionality would result in 100ms blip (500-400): a pulse is supposed to happen at 150bpm, 400 ms after the first note, but a new set of synchronic notes cannot be configured from the cluster until the clusterTimer surpasses the clusterThreshold (500ms after first note). So the first note will come 100ms late.

Mike ToDo

UI

• Preparation names, dropdown menu for selection
• Keymap keyboard toggle (BKKeymapComponent, BKKeymapState, BKKeyboardComponent, BKKeyboardState)
• Piano Construction Site UI
• Toggle

Presets (after parameter keymap)

• Reading / Writing (Preset/Library/Files)

Other

• Basic scripting math
• External controller input for parameters

NOTES/BUGS

TODO:

1. Switch pianos with key event
2. Modification map with interface
3. Save pianos to disk as xml, load them from disk (essentially Galleries)
4. Gallery interface

THOUGHTS:

• Active preparation model idea: each Preparation class has static and active copy of all fields. or there is a substruct in each Preparation class called "Active" which has another set of parameters that are active parameters

• Performance mode.

• AudioPluginDemo example has great state-saving XML stuff to learn from.

BUGS:

• (important) when transposition changed via modification and pianist is using legato fingering (ie notes are never all off), noteoffs arent registered properly. relatively easy fix. current algorithm is to count notes that are on.
• (minor) Hammers/Release samples arent transposed before they are chosen by synthesiser (so we get wacky large transposition sounds)
• adding to PianoMap with keymap (k1:2, any key in keymap 1 change to keymap 2) requires a little more intelligence. each Piano needs to know which particular keymaps are in its PianoMap. this is where a proper PianoMap class would come in handy. up til now have only implemented the PianoMap with simple integer arrays.
• (important) if transposition or gain change during playback (while note is down) from user input or modification, the noteoff will not be registered properly in the synths. should be able to solve this relatively easily.
• if you change pianos while notes are down and then keep playing notes (ie, at least one note is always down) noteoffs will still be sent to previous pianos. (traightforward fix, but more CPU intensive: need to have memory of specific notes that are held down at time of change and deal with that info).
• (minor) decreasing nostalgic length multiplier has undefined behavior
• (minor) when switching sample sets, should first send key off for notes currently pressed to all active processors

Major cleanup in order

...now that I know how to forward declare.

Nostalgic ToDo

• fix beatsToSkip issue; needs to take in to account different beat lengths. i think we need a function in Synchronic that takes NostalgicBeatsToSkip and returns the right number of samples; will do [dt]

• fix click at end of reverse playback when play rate < 1

test issue

• do thing 1
• do thing 2
• do thing 3