Audio link breaks down after some time about jacktrip-webrtc

[rezonant]

So I believe audio worklets are adaptive, if the code in the worklet takes more time than is available for processing a certain batch, the UA introduces additional buffering to compensate, so it makes sense that you would see this "slipping" behavior over time. Not sure if you already did this, but if you measure how long it is taking to process each batch and periodically output statistics, you may be able to confirm that indeed the worklet is taking too much time to handle (some) batches.

If that's the case the only remedy would be to reduce the amount of time the worklet has to run for via optimization, or otherwise accommodate the delay on the other end, but that seems counterproductive since the point is to reduce overall latency.

Interesting problem (and project!)

[matteosacchetto]

Hi Giuliano,
thanks for checking this project.

Yes, that behavior is something is already experienced during the development phase and is something I thought to have somehow solved. Well apparently it still happen sometimes.

In order to debug this I need some firther informations. Can you please provide the following informations?

PC-1
Machine details

• OS and version: ...
• Web browser and version: ...
• Power optimization settings: ...

JackTrip-WebRTC instance

• Playout buffer size: ...
• Activated speaker selection? (Yes/No)

PC-2
Machine details

• OS and version: ...
• Web browser and version: ...
• Power optimization settings: ...

JackTrip-WebRTC instance

• Playout buffer size: ...
• Activated speaker selection? (Yes/No)

The same effect doesn't seem to happen to the video, that keeps being streamed between the 2 devices.

This is related to the fact that audio and video are handled in two completely different ways. For video we are using MediaStreams, while for audio we came up with a custom solution in order to try to reduce the mouth-to-ear latency. That solution relys on Web Audio API's AudioWorklet and sends data through the WebRTC's RTCDataChannel.

Since we are using AudioWorklets we are actually creating threads in a browser in order to perform the audio processing we need. Unfortunately, there seems to be some problems in how the AudioWorklets are handled within especially Chrome (v85+), which leads sometimes to an increasing delay in the call of the process function of the AudioWorklet in charge of extracting the data from the DataChannel, which leads to deliver packets with a delay with respect to when they should have been generated. Since that delay is increasing more and more without decreasing and since the receiver filters out packets which arrive too late, what happens is that from a certain moment on all the packets generated by one of the peer are filtered out, so the peer appears as silent, but if you take a look at the upload network bandwith used by the machine you will see that it doesn't change, so it still generates and sends those packets.

It is something I still haven't figured out how to completely solve, and which possibly is due to be some sort of strange behavior of the browser. I will check further on this.

Cheers,
Matteo

[giulianobernardi]

Hi Matteo

This is related to the fact that audio and video are handled in two completely different ways. For video we are using MediaStreams, while for audio we came up with a custom solution in order to try to reduce the mouth-to-ear latency. That solution relys on Web Audio API's AudioWorklet and sends data through the WebRTC's RTCDataChannel.

Right! I even read that in some other open issue. Thanks for the clarification

Regarding the setup I'm using

Machine details

PC-1
OS and version: macOS Catalina 10.15.7
Web browser and version: Chrome Version 87.0.4280.88 (Official Build) (x86_64)
Power optimization settings: It's attached to the power adapter so there doesn't seem to be any power optimization going on

PC-2
OS and version: macOS Catalina 10.15.5
Web browser and version: same as above
Power optimization settings: same as above

JackTrip-WebRTC instance

I tried different parameters

Playout buffer size: 2-8-16-30, kept the same on both devices
Activated speaker selection? Both Yes/No

but the result always seems to be the same: the audio feed breaks down after a variable time between a bunch of sec and 1-2 min.

Later I'll try again with a Windows machine as client device.

PS: I made a few tests on my LAN both over Ethernet and over WiFi. The problem happens in both cases.

I also noticed, by monitoring the stats, that when the audio breaks down the current value of the playout buffer gets down to 0 and doesn't change anymore.

[matteosacchetto]

Ok ok, if you can try it out also in Firefox, to see if you have the same behavior or not.

From the test I performed during development I would advise to set the playout buffer to a value >= 8.

PS: I made a few tests on my LAN both over Ethernet and over WiFi. The problem happens in both cases.

Yes, because the problem is not something related to network, but it is something already present at the generation of the packets, so having or not the network will not make any difference. I will guess that you would experience the same behavior also if you open two clients on the same machine and perform the connection locally (on localhost) without involving network in the picture.

Unluckily this behavior seem to be something which sometimes pops out and sometimes vanishes, especially when there is a browser version update, and up until now I have not fully found what causes this behavior (I thought I did, but apparently it seems to not be the only cause...)

---

Suggestion on Chrome:

• Try to activate the flag "Use realtime priority thread for Audio Worklet"

Let me know if it performs any better

[giulianobernardi]

Hi Matteo.
I did a few more tests.

---

Test 1

PC-1 (a not-too-old 2018 MacBook Pro with macOS Catalina 10.15.7).
PC-2 (an old-ish 2014 MacBook Air with macOS Catalina 10.15.5).

I tried again to use Chrome upon activating the flag "Use realtime priority thread for Audio Worklet" on both PCs.
No luck. Same results as before.

---

Test 2

PC-1 (same as above): I tried a few different versions of Firefox: the one I had already installed (forgot to write it down, apologies, but it's probably from a few months ago), 84.0.1 (64-bit), and the latest as of today 84.0.2 (64-bit)
PC-2 (same as above): Firefox 84.0.1 (64-bit), and the latest as of today 84.0.2 (64-bit)

With Firefox, PC-2 doesn't even manage to read the microphone signal. I double checked and the mic access was allowed both on a system level (in the macOS Privacy settings) and at the browser levels but nothing... Quite weird as I'm not experiencing the problem on PC-1.

---

Test 3
PC-1 (same as above): Using Firefox 84.0.2 (64-bit)
PC-2: Back using Chrome Version 87.0.4280.88 (Official Build) (x86_64)

Same behaviour as in the original question (audio link breaks down after some tens of seconds)

---

Test 4
Here I changed my PC-2 with a Windows machine.

PC-1 (same as above): Using Chrome Version 87.0.4280.88 (Official Build) (x86_64)
PC-2: Windows 10 Home 1909 (18363.1256), using Chrome Version 87.0.4280.88 (Official Build) (x86_64)

The audio link in this case broke down after a longer test time. I let it run for some 3-4 min and it still kept transmitting. The audio got a bit choppy on some occasions but, at least, it was not interrupted ;)

---

Test 5
Here I changed my PC-2 with a Windows machine.

PC-1 (same as above): Using Firefox 84.0.2 (64-bit)
PC-2: Windows 10 Home 1909 (18363.1256), using Chrome Version 87.0.4280.88 (Official Build) (x86_64)

Audio broke down after some 2 mins. However, during that time, the audio link seemed to remain very good (with low RTT and only 5-10% discarded packets, vs much higher percentages of all the other setups I tried)

---

Test 6
Here I changed my PC-2 with a Windows machine.

PC-1 (same as above): Using Firefox 84.0.2 (64-bit)
PC-2: Windows 10 Home 1909 (18363.1256), using Firefox 84.0.2 (64-bit)

The audio link in this case broke down after a longer test time. I let it run for some 3-4 min and it still kept transmitting. The audio got a bit choppy on some occasions but, at least, it was not interrupted ;)

---

Overall, I'm not sure I can find any interesting pattern from my tests :D and, given the high number of variables, I'm starting to think it's hard to define some meaningful testing procedure.
Anecdotically, it seems that my old MacBook brings some problem to the game (as using the Win machine made the communication last longer) and that using Firefox on the server lead to better QoS (up to the time where the communication, once again, broke down...).
Perhaps we should try to do this and have a more standardized way to test the implementation but I'm not sure how that could be done.

[matteosacchetto]

Hi Giuliano,
thanks for taking the time to perform those tests.

Test 2

PC-1 (same as above): I tried a few different versions of Firefox: the one I had already installed (forgot to write it down, apologies, but it's probably from a few months ago), 84.0.1 (64-bit), and the latest as of today 84.0.2 (64-bit)
PC-2 (same as above): Firefox 84.0.1 (64-bit), and the latest as of today 84.0.2 (64-bit)

With Firefox, PC-2 doesn't even manage to read the microphone signal. I double checked and the mic access was allowed both on a system level (in the macOS Privacy settings) and at the browser levels but nothing... Quite weird as I'm not experiencing the problem on PC-1.

It may be due to the sample rate the microphone is set to use. In the application i am using 48khz, and if the mic is set to 44.1kz on firefox it creates an error since it does not allow to have two different sample rates set for the mic and the AudioContext (I hope it is somewhat clear... I could have explained it better 😅)... Take a look at the Firefox console on the PC-2 and tell me if you see any errors.

Regarding the problem you are describing, it is something I will have to investigate further... It is something which already happened in the past and for which apparently I still have not found a full solution, since I have not identified what may be the cause of the issue. I will try to investigate further on this and keep this issue updated with what I discover

Thanks again! Cheers,
Matteo

[giulianobernardi]

Hi Matteo
Thanks for the assistance.

It may be due to the sample rate the microphone is set to use. In the application i am using 48khz, and if the mic is set to 44.1kz on firefox it creates an error since it does not allow to have two different sample rates set for the mic and the AudioContext (I hope it is somewhat clear... I could have explained it better 😅)... Take a look at the Firefox console on the PC-2 and tell me if you see any errors.

OK. Understood. I quickly checked the console on PC-2 (the MacBook) and it does indeed give an error about sampling-rate mismatch.

I then checked my (system-wide) mic settings and it was indeed on 44.1 kHz for PC-2 (on macOS this can be done through Audio MIDI Setup, for anyone that might need to check it out 😄 ). PC-1 was correctly on 48 kHz.
Anyway, I then changed PC-2's mic to 48 kHz, but the problem persists. Not sure of what that means 😄

[matteosacchetto]

Anyway, I then changed PC-2's mic to 48 kHz, but the problem persists. Not sure of what that means 😄

Well, that's kind of strange 😅... I remember that on linux I had to restart PulseAudio and the web browser in order to make it work... Maybe something similar needs to be done with CoreAudio on the MacBook (i'm just guessing... I actually do not know 😅 )

[giulianobernardi]

I had only tried a sudo launchctl kickstart -kp system/com.apple.audio.coreaudiod + restarting the browser and that didn't do the trick.
But now I tried also to run sudo killall coreaudiod and it's working, i.e. the webRTC app in PC-2 gets an input audio stream through the mic.

I ran another couple of quick tests, with PC-2 running Firefox and PC-1 running once Chrome and once Firefox and it's going well in both cases, i.e. the audio feed is kept throughout a 4-5 min test. Latencies also seem to be much smaller.

Perhaps, in the tests I did this morning, the need resampling for resampling was screwing the stability of the app even in Chrome.

[matteosacchetto]

Perhaps, in the tests I did this morning, the need resampling for resampling was screwing the stability of the app even in Chrome.

Probably the overhead introduced by the resampling was contributing to those incremental delays and so to the bad audio experience

[matteosacchetto]

Hi @rezonant,
thank you for the suggestions!

I will check with Chrome profiler and then parse those information to have the details about how much time it takes to process each batch. This way we are able to see if what happens is what yo are describing, or if it is something else. Thanks 😄

[artit91]

It seems to me that using a scriptprocessornode with a 1024 buffer, send it through the network and play it as an audiobuffer immediately gives the same result as your app because maybe your processing uses no buffers but the playout needs one and the processor uses one under the hood and flushes it on each call of process.

Btw the 1024 buffer is 20ms latency and opus operates on 3-20ms if you override the sdp.

Sending the data in 20ms chunks and play them is a simple and working compromise. I'm pretty confident that a network buffer (with forward correction) should be used instead of a playbuffer as mediastream operates by default.

I think the issue is related to the same issue why the scriptprocessor node doesn't work properly (has a lot of jitter and silence) under 1024 buffersize.

[matteosacchetto]

Hi @artit91,
thank you for the feedback.

In the standard configuration of the application we are using a playout buffer of 8 packets (128 samples per packet), so it is correct that using a ScriptProcessorNode with a buffer of 1024 samples you get the same result 😄.

Anyway, I am not completely sure about this:

I think the issue is related to the same issue why the scriptprocessor node doesn't work properly (has a lot of jitter and silence) under 1024 buffersize.

Since, AudioWorklet have been desinged to use a 128 sample buffer and to run on a separate audio thread, differently from the ScriptProcessorNode

Cheers,
Matteo

[artit91]

Hi @artit91,
thank you for the feedback.

In the standard configuration of the application we are using a playout buffer of 8 packets (128 samples per packet), so it is correct that using a ScriptProcessorNode with a buffer of 1024 samples you get the same result 😄.

Anyway, I am not completely sure about this:

I think the issue is related to the same issue why the scriptprocessor node doesn't work properly (has a lot of jitter and silence) under 1024 buffersize.

Since, AudioWorklet have been desinged to use a 128 sample buffer and to run on a separate audio thread, differently from the ScriptProcessorNode

Cheers,
Matteo

Me neither, I was drunk sorry. :D

It would be nice to know how many context switches happen in inter-thread communication. It really can kill the performance leading to the behaviour observed.

[matteosacchetto]

It would be nice to know how many context switches happen in inter-thread communication. It really can kill the performance leading to the behaviour observed.

That's an interesting point! Maybe with browser's profiler we can somehow obtain that info.

Thank you for the feedback :)
Matteo