robomex / reality-synthesizer

iOS 15.4 AVFoundation LiDAR sensor + MIDI notes = depth-based augmented reality music visualization as the music is being played

Reality Synthesizer: Visualize MIDI Notes in Augmented Reality

iOS 15.4 opened up the LiDAR sensor to AVFoundation. In this demo MIDI notes played on a paired synthesizer are interpreted by Metal shaders to produce augmented reality visualizations. This demo builds on parts of Capturing Depth Using the LiDAR Camera, AudioKit, and Linnstrument Helper.

The demo with sound can be viewed at this tweet.

reality-synthesizer's Issues

Notes get "stuck"

Sometimes, typically when releasing multiple keys on the connected synthesizer at the same moment, SwiftUI's .onChange(of:) is called on the same frame – resulting in one of those calls not being executed. This results in a note's visualization to get "stuck on" even though the key is no longer being pressed and the note is no longer being played.

The "I'm going to ignore this" fix in place is the stuck key will be released on a subsequent release of that same key.