SFU Server: Multiple users connected simultaneously, implement view-only password, isolated sessions about selkies-gstreamer

[ehfd]

@nums Thank you for the information! We are currently focusing on improving the documentation for more contributor accessibility. We have a goal for this project to potentially attract community contributors from every relevant field, from experienced GStreamer/WebRTC/conferencing/media/graphics software developers to video power users who simply know a lot of about optimal encoder parameters, and Python/JS learners.

[ehfd]

Relevant code: https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/signalling_web.py
Should be addressed with #27.
The signalling server must be rewritten to accommodate multiple peers. Anyone with experience in WebRTC development can do it.

[ehfd]

This might be addressed with #48.

[nums]

To be able to send stream to others participants, I think the ideal way is to send stream to a WebRTC SFU server.
Neko project is using Pion WebRTC (https://github.com/m1k1o/neko) to achieve that.
Webrtcbin must be able to interface with SFUs (there is a WHIP client that uses webrtcbin https://github.com/meetecho/simple-whip-client)
Once done, the main problem is the management of the quality of the streaming with the recovery of Nack events, PLI etc and this can produce lags or a noticeable drop in the quality of the video.

The mediasoup documentation explains this very well:
https://mediasoup.org/documentation/v3/scalability/#one-to-many-broadcasting

[ehfd]

https://github.com/centricular/webrtcsink instead of webrtcbin may solve this issue as this has its own signalling server.

[nums]

It can, but they incidate that :

It is important to note that at this moment, encoding is not shared between consumers. While this is not on the roadmap at the moment, nothing in the design prevents implementing this optimization.

So it is possible, but there will be a big use of resources. Some certain nvidia card have a limitation of number of encodings so it can be a problem.

I'm working on my side to use webrtcbin with a SFU with this "bridge" https://github.com/meetecho/simple-whip-client and to stream the desktop screen to an SFU like mediasoup. I just have a little issue on the compilation with the selkies gstreamer version, but "work in progress"

[ehfd]

@nums I guess that you are working on a contribution. We greatly thank your time.

[nums]

Yes I'm trying to share the screen in "readonly" mode, but to be able to take the control il will be an other story

[ehfd]

Yes I'm trying to share the screen in "readonly" mode, but to be able to take the control il will be an other story.

That's already something we really need. Thanks a lot!

[ehfd]

Relevant: #48 (comment)

[ehfd]

WebRTCSink is a wrapper for WebRTCBin, where settings are available in https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/blob/main/net/webrtc/src/webrtcsink/imp.rs

The issue of WHIP/WHEP is that there is no DataChannel for keys, cursors, and other information. Only read-only access would be possible without other transport protocols.

Thus, the immediate property might rather be a multi-user SFU server.

Need contribution for the SFU server, perhaps utilizing GStreamer's own signalling server.

https://gstreamer.freedesktop.org/documentation/rswebrtc/index.html#usage

[PMohanJ]

The current architecture of selkies-gstreamer consists of two webrtc connections(video and audio) for one single session to a user. So using plugin webrtcsink would result in 2*n connections (of course, all are handled by webrtcsink itself) for n number of session?

If so, considering the below note, wouldn't it be resource intense even for as little as 2-3 users?

It is important to note that at this moment, encoding is not shared between consumers. While this is not on the roadmap at the moment, nothing in the design prevents implementing this optimization.

[ehfd]

So using plugin webrtcsink would result in 2*n connections (of course, all are handled by webrtcsink itself) for n number of session?

This is called an MCU server.

An SFU server distributes the same stream(s) to many clients. I still think webrtcsink's signaling server is still capable of doing this.

https://forasoft.medium.com/p2p-vs-mcu-vs-sfu-what-to-choose-for-a-video-conference-app-fa5cb4a5dd6e