Videobridge considers endpoints with muted video when building lastN list about jitsi-videobridge

[bgrozev]

It was probably introduced in early 2022 with the multi-stream feature. The bug only triggers when multi-stream is enabled, and it was enabled by default in on June 28 (#1912). I think the thread you linked to has multiple confounding issues (some of them fixed back in 2020/2021), but it's possible some of the more recent reports are related.

[bukharin]

@bgrozev this patch fixes the problem 🎉

[bgrozev]

I agree with your expectation, but wasn't able to reproduce the problem in a unit test (and I don't think it's easy to simulate with jitsi-meet either). It would be helpful if you can give a way to reproduce with jitsi-meet, or in a unit test. See my attempt here:
https://github.com/bgrozev/jitsi-videobridge/tree/audio-only-test

[bukharin]

@bgrozev Ok I will try to reproduce it soon.

I would like to clarify the case: it reproduces only if endpoint C joins with video track and mutes them later. If it joins in audio only mode - JVB works as expected

[bukharin]

@bgrozev in your attempt you create audio only participany with empty array of MediaSourceDesc.

I guess that in my case Endpoint has muted MediaSourceDesc in mediaSources list and getEffectiveContraints ignores this to determine which sources sould be forwarded

[bukharin]

@bgrozev I found a way to reproduce the problem on meet.jit.si:

1. Open 3 tabs with url: https://meet.jit.si/blablabla#config.#config.channelLastN=1. All tabs will ask JVB to receive lastN=1
2. Join on all tabs with active cameras and muted mic
3. Switch to tab 2, unmute mic and say something to set dominant speaker
4. Tab 1 receive from JVB (meet.jit.si uses source signaling and messages are different, but it does not matter)

{"colibriClass":"ForwardedSources","forwardedSources":["%tab-B-source-name%"]}

Correct.

5. Mute mic on tab 2, switch to tab 3, mute video and unmute mic (to set new dominant speaker).
6. Tab 1 receives empty forwarded sources:

{"colibriClass":"ForwardedSources","forwardedSources":[]}

PS: I am using Chrome devtools to inspect JVB bridge channel messages

[bgrozev]

@bgrozev I found a way to reproduce the problem on meet.jit.si:

I can also reproduce on meet.jit.si. After testing a few versions I think I somehow accidentally fixed it in #1963. 🤦

Can you check if you see the problem on beta.meet.jit.si? Does the problem still occur in your application with jvb after #1963? Note that your app needs to notify jvb that a source was muted by sending something like this (like jitsi-meet does)

{"colibriClass":"SourceVideoTypeMessage","sourceName":"1d98d1eb-v0","videoType":"none"}

Edit: I think I see why it was broken: ConferenceSpeechActivity used Endpoint.videoType which was not updated when a source's video type changed.

[bukharin]

@bgrozev I can confirm that problem not reproduces on beta.meet.jit.si and in 2.2-57-g1158321d version. Thank you!

There is an understanding how long this problem exists? It may be part of long standing issue with inactive status, because JVB not forwarding video participants in priority of participants with muted video.

[bukharin]

@bgrozev I found similar behaivour in JVB with enabled multistream mode. It occurs when one of videostreams of participant goes to muted state.

One of participants sends 2 streams (desktop + camera).. Remote participant ask 3 lastN

{
   "colibriClass":"ReceiverVideoConstraints",
   "lastN":3,
   "defaultConstraints":{"maxHeight":180},
   "constraints":
   { 
      "55952e51-v1":{"maxHeight":2160},
      "55952e51-v0":{"maxHeight":180}
   },
   "onStageSources":["55952e51-v1","55952e51-v0"],
   "selectedEndpoints":[]
}

JVB asnwers correct and forward 3 last-n video streams

{
  "colibriClass":"ForwardedSources",
  "forwardedSources":["55952e51-v1","55952e51-v0","ad0f89aa-v0"]
}

If I mute 55952e51-v0 source, JVB start sending only 2 last-n streams, but conference has more participants with unmuted videos:

{
   "colibriClass":"ForwardedSources",
   "forwardedSources":["55952e51-v1","ad0f89aa-v0"]
}

[bgrozev]

Thanks for the details, I think I have a fix. The code which moves the disabled streams down the list (ConferenceSpeechActivity) works at the endpoint and since your endpoint hasVideoAvailable it stays high in the list. We then just concatenate all endpoints' sources which leaves a disabled source high in the list.

Would you be able to test #1989?