Potential memory corruption when receiving dense event streams about midiapps

[krevis]

I just released MIDI Monitor 1.5 which has this fix.

[krevis]

How fast is "extremely rapid"? Is there an easy way I can run this app (which app is it?) to reproduce the problem? I can definitely give it a try.

[orchetect]

I'm firing multiple single-event packets as fast as possible from a custom MIDI library to MIDI Monitor using the old CoreMIDI API. Back to back synchronously in the same function call. The library can receive its own events in such fashion without issue, and also send to other applications without issue. That is why it appears that MIDI Monitor may have an issue receiving such messages.

Often it's literally only 2 packets that are immediately back-to-back and many times MIDI Monitor receives partially correct data but partially garbage data. It's random. Seems like some sort of pointer or memory corruption. I ran into similar issues when testing the new Core MIDI API.

[krevis]

OK, so just to make sure, is this what you're doing?

• Your app makes a CoreMIDI virtual source
• In a loop in a single thread, you do this repeatedly:
• • Create a MIDIPacketList with a single MIDIPacket with a single MIDI message (e.g. a 3-byte note on or note off message), with a zero timestamp (meaning "send immediately")
• • Call MIDIReceived() to send that packet list to the virtual source

(Or are you setting MIDI Monitor to "act as a destination for other apps", so it makes a virtual destination, and then you use MIDISend to send to that destination port?)

[orchetect]

Yes exactly. Create a virtual source, tick that source in MIDI Monitor under "MIDI sources". Not using the 'spy on output' feature.

[orchetect]

I also observed the same behavior when MIDI Monitor receives data from Pro Tools (same system as mentioned above).

If Pro Tools is set up to send HUI data to MIDI Monitor, it will sometimes send burst data where there are multiple packets sent in immediate succession, and MIDI Monitor randomly shows some events as garbage data.

(Which means currently, MIDI Monitor can't be relied on to display accurate data, at least on this system OS/architecture combination, and possibly others.)

[krevis]

So far I haven't been able to repro with my own test app. (Attached: SendMIDIDataFast.zip. Build, run, and use the menu commands under SendMIDIDataFast to send just one packet, a bunch of packets all at once, or a bunch of packets repeatedly.)

This seems to be working w/o corruption on my M1 MacBook Pro, Monterey 12.0.1, with both MIDI Monitor 1.4.1 (old ObjC code) and the current main branch (Swift rewrite).

If you could give me some way to run your code to generate events, it would save me a great deal of guesswork. Happy to do it over email. I unfortunately don't have Pro Tools.

[orchetect]

Thanks for putting that together.

I noticed a few things when tinkering with parameters. Using the "Send A Bunch" menu item:

• Your test code, as-is, arrives at MIDI Monitor correctly
• If timeTag is set to a high number (ie: mach_absolute_time(), such as 10345550615470):
  • MIDI Monitor receives mostly garbage data, and also hangs with a beachball for around 10 seconds during it
• If timeTag is set to a VERY high number (ie: 431039240816666, a real-world timetag from my Yamaha keyboard USB drivers):
  • MIDI Monitor receives no data at all; the incoming event history is empty

[krevis]

Aha, I didn't remember that a zero timestamp is not technically correct to give MIDIReceived(), according to the documentation. I can repro the same problem, if I just set the timestamp to AudioGetCurrentHostTime(), which is more correct anyway. Thanks for the help, investigating.

[orchetect]

Yes, just checked the docs on MIDIReceived:

Unlike MIDISend(), a timestamp of 0 is not equivalent to "now"; the driver or virtual source is responsible for putting proper timestamps in the packets.

Anecdotally, I also recall on very rare occasion, some software (such as DAWs, which care about timing information) would not recognize MIDI events that had a timetag of 0, depending on their particular implementation. However this was exceedingly rare as most applications ignore the timetag entirely.

AudioGetCurrentHostTime() just returns mach_absolute_time(), FWIW.

[krevis]

OK, this is my fault, and (AFAIK) it is only broken on M1 machines.

When CoreMIDI gives us a MIDIPacketList we were incorrectly computing its size if there was more than one packet. On M1 (ARM) there may now be up to 3 bytes of padding between subsequent MIDIPackets. (On PowerPC and Intel they were packed tightly together.) As a result some data at the end of a packet list could get corrupted.

This is already fixed on the main branch, because during the rewrite in Swift, I switched to MIDIPacketList.sizeInBytes() and SMPacketListSize() which doesn't have that bug.

Now to decide whether I should put out another release from the old ObjC branch, or just take this as a cue to finally finish up the Swift branch and release it.

[orchetect]

Ok interesting. May be architecture memory alignment semantics which could be obfuscated behind their size method so you'd be lucky if any of that was mentioned in the documentation.

[krevis]

It's documented in the header.... well, tersely, but it is there:

	#if TARGET_CPU_ARM || TARGET_CPU_ARM64
		// MIDIPacket must be 4-byte aligned
		#define MIDIPacketNext(pkt)	((MIDIPacket *)(((uintptr_t)(&(pkt)->data[(pkt)->length]) + 3) & ~3))
	#else
		#define MIDIPacketNext(pkt)	((MIDIPacket *)&(pkt)->data[(pkt)->length])
	#endif

If I had ever used CoreMIDI on iOS devices (which have always been ARM) I would have noticed the ramifications 10 years ago. Oh well.

[orchetect]

Makes sense, as I've seen many StackOverflow threads over the years about Core MIDI working in unexpected/inconsistent ways on iOS. I'm sure this played some part.