[idea] Use wasm in web workers about jacktrip-webrtc

[matteosacchetto]

It is as simple as: ffmpeg -i <my_video>.<mp4|mkv> <output_audio>.wav

Just to clarify, this setup works, but it is a bit janky... It requires a lot of human involvement in the process

[matteosacchetto]

Taking into consideration that the Processors in the web workers add enough latency to make network performances impractical, (according to the measurements on @matteosacchetto dissertation)

Wait, when you say that, what exactly are you referring to?

[fredguth]

If I understood correctly, table 4.8 shows a median Latency of 55 ± 3.3 ms, while for network performances we would want sub 30ms or 20ms. And that, I believe, is just for capturing and playing the audio, it does not include the network latency (more 35ms for a US east - west coast, for example https://serverfault.com/q/137348 Pehaps if the distance is not so huge it can be sub 10ms, idk).

Anyway 55 ms is already sub optimal for network performance, right?

[matteosacchetto]

55 ms is suboptimal you are right, but let me add a comment to clarify all the aspects

As you already said, in order to reach results which are good in a NMP scenario we should be sub 30ms, at least on the whole processing chain.

If we consider that result, removing ~22ms of latency due to the internal de-jitter buffer (8 packets of at 48kHz), we obtain that the 55ms become 33ms of processing. If we compare this result with table 4.4 (Windows - Chrome - Configuration 1), we can see that we are actually adding 6ms due to processing. This is still not that true, since, we need to factor in the fact that latency due to the/restart of the audio engine of the OS has some terms which are contained within those 6ms, and that those 6ms also include the transmission through the RTCDataChannel, we can say that the actual processing due to the use of AudioWorklet is in the rage 0 - 5 ms. Delay which is mostly due to the use of MessageChannels for exchanging data between execution contexts. Additionally, we can also factor in those 5ms of variance the additional load put on the system, since the measurements in my thesis, due to physical constraints related to the pandemic, were performed with only 1 device, using OBS to record the direct output and the output of the app.

So, I would not say that the latency which we see is due to the AudioWorklet, since they are really efficient! As we can see from table 4.4, most of the latency is already introduced by the audio acquisition/playback (nearly 30ms on Windows with the internal audio card)

To conclude, I do not see the use of WASM that relevant with respect to AudioWorklet, since AudioWorklet are designed for real time processing and can take advantage of being implemented directly within the browser

[fredguth]

Ah... I see. I read in your dissertation that you used OBS to record the loopback and measure the time it took, but I still didn't get if you did it with a video tool or built something to check the loopback time in the video.

Have you tested with Opus? Does it reduce too much the sound quality?
Does sending audio-only (no video) change the latency somehow?

[fredguth]

I decided to close the issue as it really doesn't seem to be something valid about investing time in this effort.

[matteosacchetto]

Just to reply to some of your questions

Ah... I see. I read in your dissertation that you used OBS to record the loopback and measure the time it took, but I still didn't get if you did it with a video tool or built something to check the loopback time in the video.

We used OBS to record an impulsive sound, like a finger snap, both directly form the input mic and at the output of our application. We then extracted using ffmpeg, the audio from the video we obtained through OBS. Given that the audio was at 48kHz, we then analyzed the distance between the two waveform (direct and played back by our app) and evaluated latency that way

Have you tested with Opus? Does it reduce too much the sound quality?

We decided to not thinker with Opus, because our goal was to send uncompressed audio and reduce latency, so any kind of compress is worse in term of latency with respect to no compression

Does sending audio-only (no video) change the latency somehow?

No significant differences were observed.

Matteo

[fredguth]

The ffmpeg extraction was just a command? Or a script? Is it in this repo? I want to reproduce your research but doing many loops to help with precision.