Call for Feature Requests about mediasoup

[ldenoue]

As mentioned in the first idea, I vote for Recording streams too.

[justindujardin]

We've been eagerly tracking this work and considering it for use in the future.

Here are a few requirements that we have for our existing SFU (I haven't compared them to what you've built here):

• Data channel support for sending high frequency conference specific data.
• Stream level control for managing individual streams for a bundled peer connection.
• Conference/peer stats for monitoring and health check purposes.
• Trickle Ice for quick peer connection setup times (very important to our use-case.)

---

While it's a bit more touchy, I'd love to see your Javascript API written in Typescript. It is so great that it even brings Microsoft and Google together.

You can still export an ES6 API with no types for your hardcore JS users that install the package, and you can automatically provide the benefit of enhanced code completion and type safety to Typescript users that consume your library.

I'd be happy to contribute to any Typescript efforts if you are so inclined.

[mccob]

Not sure if it's the philosophy of mediasoup: ability to record on server all the streams of a room ? (for playing them after)

[ibc]

What are your thoughts on audio mixing in order to connect to a softswitch like Freeswitch or Asterisk?

Honestly zero thoughts. mediasoup won't mix audio/video streams just to make legacy SIP endpoints happy. This is a SFU. Legacy telephony support is not in the feature list.

[ibc]

[justindujardin]

@ibc thanks for the information. After a little RTFM I see that you opt out of the higher level things like signaling. I think that is a really nice decision.

I understand the hesitation with Typescript. There are some lower impact options like crafting a single mediasoup.d.ts file that describes your library API types. They're easy to produce provided you have a simple API (which you do) and would be easy to maintain. I'll bring it up again after 1.0.0

It seems that with mediasoup will provide most of what we need once you support data channels. I'll stay tuned. Thanks for your effort. 👍

[ibc]

After a little RTFM I see that you opt out of the higher level things like signaling. I think that is a really nice decision.

mediasoup is not a "SFU built on Node.js" but a "SFU for Node.js" :)

There are some lower impact options like crafting a single mediasoup.d.ts file that describes your library API types. They're easy to produce provided you have a simple API (which you do) and would be easy to maintain. I'll bring it up again after 1.0.0

Great!

[ibc]

but receiving all streams gives us this ability. Same goes for detecting who the active speaker is.

That's a (good) use case, but not _The Use Case_©️.

Now imagine a social multiconference application in which anyone can see the video of all the others, but just those who pay some money are able to send and receive the audio of other participants. You don't want to mute the audio in client side.

[ibc]

http://mediasoup.org/faq/#is-mediasoup-a-native-addon

• I don't depend on internals of Node.js or V8 (which are not stable as your previous link clearly shows).
• I can run many workers in parallel.
• In the future, if Node.js disappears (ough) I could connect mediasoup-worker to any other high level runtime.

[notedit]

there is still no example to test mediasoup, can you add one?

[ibc]

@mccob the API already allows handling received RTP packets in JavaScript land:

• rtpReceiver.on('rtpraw')
• rtpReceiver.on('rtpobject')

Whether having the raw RTP packets in JS is enough to playback them later is another topic that should be done by another library. But yes, I will work on that.

@notedit Usage examples will be properly written once the API is finished. But now I'm asking for "features" :)

Thanks a lot.

[ibc]

@justindujardin thanks for your feedback. Comments:

Data channel support for sending high frequency conference specific data.

DataChannel will be supported in the next version (after 1.0.0). In fact, I plan two modes:

• DC messages are directly routed to every room participant with DC support.
• DC messages reach the application JavaScript land, so the app can handle them and decide what to do with them.

Stream level control for managing individual streams for a bundled peer connection.

I don't fully understand what you mean with "stream level control", but take into account that BUNDLE is a SDP artifact because, originally, SDP did not allow two media streams over the same transport.

ORTC, and hence mediasoup, lets the app create as many Transports as desired, and attach them to RtpReceivers also as desired.

Conference/peer stats for monitoring and health check purposes.

Yep, there will be some kind of API for that.

Trickle Ice for quick peer connection setup times (very important to our use-case.)

mediasoup is not SDP based so talking about "Trickle ICE" has no sense here. You create a Transport for a remote Peer and retrieve its local ICE candidates. How you pass them to the remote Peer (browser) is up to you.

Regarding TypeScript, I don't want to force all the library users to deal with TypeScript. But I'm not against providing an extra TypeScript layer (probably within a separate library that would act as a mediasoup TypeScript wrapper). Let's continue with this topic once 1.0.0 is out :)

[peili]

What are your thoughts on audio mixing in order to connect to a softswitch like Freeswitch or Asterisk?
As far as I know they don't support the transport of multiple audio streams on the same peerconnection yet. (Correct me if that's wrong)

[oteku]

Some ideas

• Provide client SDK (js / Android / iOS) should simplify adoption of mediasoup
• Provide a plugin system
  -> a video recorder plugin will be nice
  -> a computer vision plugin based on OpenCV (face recognition, sentiment analysis, image overlays, ...)

[saghul]

a computer vision plugin based on OpenCV (face recognition, sentiment analysis, image overlays, ...)

This one would require that MediaSoup decodes the video and analyzes it, which can get really CPU intensive. I'm not sure what the long term goals are, but in the short term @ibc wants to build a SFU, that means no decoding of the video... Maybe it could be added later on though or maybe find a way to integrate it nicely with Kurento which already does some of that IIRC.

[ibc]

Provide client SDK (js / Android / iOS) should simplify adoption of mediasoup

This won't happen. mediasoup is just a "piece" of the puzzle. It does not provide a network signaling protocol, so it makes no sense to provide a client SDK. Also, mediasoup is just about WebRTC (media plane). Clients don't need special stuff other than a proper usage of the WebRTC API.

"Products" based on mediasoup may provide a client SDK, but mediasoup itself is "just" a server side component.

a video recorder plugin will be nice -> a computer vision plugin based on OpenCV (face recognition, sentiment analysis, image overlays, ...)

RTP recording is the the readmap (in fact, RTP packets can already be reached at JS layer, so the app can store/process them as desired).

[mirkobrankovic]

Maybe stream manipulation (like for Admin mode). to mute/unmute audio/video streams, kick also, on API requests

[saghul]

You mean for avoiding the forwarding of RTP? This could be done in the client, since it has access to all streams.

[ibc]

Maybe stream manipulation (like for Admin mode). to mute/unmute audio/video streams, kick also, on API requests

There will be a very simple API to pause/resume streams, on both the receiver and on each sende, something like:

• rtpReceiver.pause()
• rtpReceiver.resume()
• rtpSender.pause()
• rtpSender.resume()

There is no "admin mode". The "administrator" is the server developer/application. He decides what mediasoup does.

@saghul said:

You mean for avoiding the forwarding of RTP? This could be done in the client, since it has access to all streams.

Certainly there is good rationale to do it via API in the server rather than relying on each client.

[saghul]

Certainly there is good rationale to do it via API in the server rather than relying on each client.

I don't disagree. However, I found that many of the traditional admin-like things get simplified in a SFU scenario. Take for example muting a participant which is making noises. Instead of administratively muting them, each connected client can do it. It doesn't mean there shouldn't be another way to do things, but receiving all streams gives us this ability. Same goes for detecting who the active speaker is.

[ibc]

@saghul remember that this is not just about boring enterprise meetings ☺️☺️☺️

[saghul]

@ibc Sure, that's indeed a great counter example! 👍

remember that this is not just about boring enterprise meetings

🕴 🕴 🕴

[davedoesdev]

Would https://github.com/nodejs/abi-stable-node affect your design decision to have subprocesses which handle the media layer?

[ibc]

Would https://github.com/nodejs/abi-stable-node affect your design decision to have subprocesses which handle the media layer?

Why? mediasoup is not a Node native module.

[davedoesdev]

What is the reason to have separate worker processes as opposed to buliding https://github.com/ibc/mediasoup/tree/master/worker/src into a native module?

[davedoesdev]

Nice! Also, if means you could run certain workers at a higher priority (say if certain users pay more in the application, just for example).

[Globik]

Excuse me, is The Real Time Streaming Protocol a part of mediasoup?

[ibc]

Excuse me, is The Real Time Streaming Protocol a part of mediasoup?

For now not.

[notedit]

i am thinking some kind video file streaming or rtp input api.

so we can custom the input media source.

[ibc]

i am thinking some kind video file streaming or rtp input api so we can custom the input media source.

Definitely yes. That will be properly implemented.

[notedit]

support "Unified Plan" for multistreams when it is completed.

https://bugs.chromium.org/p/chromium/issues/detail?id=465349

[ibc]

@notedit mediasoup already supports Unified Plan.

[kolychev]

Provide a kind of "requestFullFrame" in video rtpReceiver API to send FIR RTCP request to browser.
Use case:
We implement SFU with recording of streams by capturing and storing rawrtp packets into file.
Currently if stream without lots of video changes, we have to wait too long for next full intra frame generated by browser.
By having this kind of API, there will be less waiting time between video recording command received and first frame actually recorded.

Another option is to have full control of sending any RTCP packet from node.js side, but this seems to be more complex task.

[notedit]

@kolychev mediasoup really support this . https://groups.google.com/forum/#!topic/mediasoup/XrLFm61ndcE

[ibc]

Not exactly. Indeed mediasoup sends a PLI to he corresponding browser/endpoint when an event listener is set for the rtpraw or rtpobject event. However is not possible to request sending a PLI via API.

We are working actively on mediasoup 2.0, so will incorporate such a feature in there.

[jnoring]

+1 to datachannel support.

What I'd really like to see is signaling done via regular old HTTPS (or whatever), and then the room messages handled entirely over in-band data channel. The big issue with having room/stream management happen over the signaling channel is it creates ambiguity when one fails but the other doesn't.

The more I work with SFUs, the more I think it's extremely important to minimize the number of connections established to the media server (i.e. multiplex as much as possible, so bundle/rtcp-mux/datachannels for room signaling, etc.)

[ibc]

@jnoring:

mediasoup v1 does not provide a signaling mechanism. It's up to the application to decide how to transmit SDPs between browsers and mediasoup.

mediasoup v2 comes with a mediasoup-client SDK library that, again, does not provide a signaling protocol (HTTP, WebSocket or whatever) but just emits events with message payloads that the client app must transmit to the server and vice-versa. You can see an example usage in the README: https://github.com/versatica/mediasoup-client

[jnoring]

@ibc I'm aware that mediasoup does not come with a signaling mechanism. I'm also aware that mediasoup does not have data channel support. Lack of data channel support precludes the architecture I described in my last post, which is why that feature is important to me.

[jnoring]

Also, to be clear, I like that media soup doesn't have a signaling layer. But my beef is most MCU/SFU implementations recycle the signaling layer for stuff that should be multiplexed over a data channel (e.g. join room, room events, application messaging, etc.)

Example: licode uses socket-io for both signaling and room/application events.

[ibc]

@jnoring still I don't get your point. If licode uses socket.io that means that licode provides a signaling layer. mediasoup does not. But you can use socket.io for both, sending mediasoup API requests/responses/notifications (in mediasoup v2) and sending your custom app messages (such as chat ot whatever you wish).

[jnoring]

@ibc the whole point is not to recycle a signaling layer for "custom app messages", because that creates duplicate persistent connections. In any event, it isn't important--what's important is data channel support.

[ibc]

the whole point is not to recycle a signaling layer for "custom app messages"

In mediasoup v2 it would be the other way around: "recycle the app custom signaling layer for mediasoup API message".

because that creates duplicate persistent connections

If there is a single signaling layer (up to the app) why would it create duplicate persisten connections?

In any event, it isn't important--what's important is data channel support.

It will be implemented when possible. Currently the only SCTP implementation in C (usrsctp) is multi-thread, so no valid within the mediasoup design (which is multi-process but not multi-thread).

[ibc]

Please @chetanbnaik, use the mailing list for questions and not a random issue in Github.

[ivatsa]

Ability to chain multiple SFU instances for oneway realtime streaming (low-latency broadcast).

PS: Did not see any new call for features after 2.0.0, hoping this is still a right place to ask for features.

[ibc]

Ability to chain multiple SFU instances for oneway realtime streaming (low-latency broadcast).

That's in the radar, but will require some time.

[ivatsa]

Cool looking forward to it.

[ivatsa]

Is there any timelines set (or version) for chaining SFU? Don't mean to demand or push for it just curious.

[ibc]

Chaining SFUs is in the radar but not yet planned. It's a MUST and will be done eventually.

[ivatsa]

Thank you so much for your prompt response

[ldenoue]

Just found this other webrtc library in python that allows live video frame manipulation https://github.com/jlaine/aiortc#aiortc

[ruslanfraps]

Maybe stream manipulation (like for Admin mode). to mute/unmute audio/video streams, kick also, on API requests

There will be a very simple API to pause/resume streams, on both the receiver and on each sende, something like:

• rtpReceiver.pause()
• rtpReceiver.resume()
• rtpSender.pause()
• rtpSender.resume()

There is no "admin mode". The "administrator" is the server developer/application. He decides what mediasoup does.

Hello, is it implemented?
Can I mute other user?
I saw there is roomClient.muteMicro(), is there the same function for mute other peers exept of me?
Also i saw audioMuted which depend of state.me, i think that i can set this props for my needs, but may be there is a better solution?
I mean mediasoup-demo

[ibc]

I mean mediasoup-demo

This is a "call for feature requests" for mediasoup, and not for mediasoup-demo.

In mediasoup you can do all you asked above. No, I won't implement them in the demo. You can do it in your application.

[information-security]

Chaining SFUs is in the radar but not yet planned. It's a MUST and will be done eventually.

Hi,

Is what described in this link, actually the documentation regarding Chaining SFUs? If not, can you please direct me to the corresponding documentation around it?

P.S.
Thank you for such a great library.