versatica / mediasoup

Cutting Edge WebRTC Video Conferencing

mediasoup v3

Website and Documentation

• mediasoup.org

Support Forum

• mediasoup.discourse.group

Design Goals

mediasoup and its client side libraries are designed to accomplish with the following goals:

• Be a SFU (Selective Forwarding Unit).
• Support both WebRTC and plain RTP input and output.
• Be a Node.js module or Rust crate in server side.
• Be a tiny TypeScript and C++ libraries in client side.
• Be minimalist: just handle the media layer.
• Be signaling agnostic: do not mandate any signaling protocol.
• Be super low level API.
• Support all existing WebRTC endpoints.
• Enable integration with well known multimedia libraries/tools.

Architecture

Use Cases

mediasoup and its client side libraries provide a super low level API. They are intended to enable different use cases and scenarios, without any constraint or assumption. Some of these use cases are:

• Group video chat applications.
• One-to-many (or few-to-many) broadcasting applications in real-time.
• RTP streaming.

Features

• ECMAScript 6/Idiomatic Rust low level API.
• Multi-stream: multiple audio/video streams over a single ICE + DTLS transport.
• IPv6 ready.
• ICE / DTLS / RTP / RTCP over UDP and TCP.
• Simulcast and SVC support.
• Congestion control.
• Sender and receiver bandwidth estimation with spatial/temporal layers distribution algorithm.
• Data message exchange (via WebRTC DataChannels, SCTP over plain UDP, and direct termination in Node.js/Rust).
• Extremely powerful (media worker thread/subprocess coded in C++ on top of libuv).

Demo Online

Try it at v3demo.mediasoup.org (source code).

Authors

• Iñaki Baz Castillo [website|github]
• José Luis Millán [github]
• Nazar Mokynskyi [github]

Social

• Twitter: @mediasoup_sfu

Sponsor

You can support mediasoup by sponsoring it. Thanks!

License

ISC

mediasoup's Issues

Two same files in different directories: for what?

Hi, trying to compile mediasoup on the windows machine and wondering, what for needing two the same files in different directories. One is https://github.com/ibc/mediasoup/blob/master/worker/src/Channel/UnixStreamSocket.cpp
And one is https://github.com/ibc/mediasoup/blob/master/worker/src/handles/UnixStreamSocket.cpp
Now unresolved symbols I have about unixstreamsocket::Close(void)

Check Fingerprint value syntax

Currently mediasoup generates Fingerprint values in hexadecimal uppercase:

SHA-1   fingerprint: 11:BB:5F:BE:48:18:5A:6C:87:D2:55:5F:4D:88:3A:DB:25:84:50:6E +0ms
SHA-224 fingerprint: 87:06:5A:7E:9E:92:2A:F2:29:62:66:01:E2:BA:E1:5A:69:D8:DC:A8:F6:5B:8E:42:A3:17:ED:5B +0ms
SHA-256 fingerprint: B0:9E:7C:B9:9C:B8:79:BB:CC:9A:3C:C5:8C:06:73:D2:D8:F0:92:99:7A:D0:BA:3C:1A:09:FF:18:9D:ED:7F:D4 +0ms
SHA-384 fingerprint: E1:47:23:6A:C3:E5:F2:94:1E:E7:30:F2:9B:7E:F7:5C:A4:57:57:33:22:E9:56:02:0B:BB:15:D4:77:1D:A2:A9:2E:E7:DA:49:22:0E:99:65:5E:38:CD:8C:37:96:CC:34 +0ms
SHA-512 fingerprint: 26:14:BD:0E:4B:1B:BD:75:6D:FF:67:70:96:03:44:A3:1A:50:24:07:30:8B:A5:51:08:7A:6D:26:E3:EE:8D:51:11:CF:6E:BC:DC:A4:20:6A:07:BB:C8:4C:87:8C:75:AC:03:5D:67:A2:39:8B:19:91:F0:88:CB:74:AC:DE:E8:FF

However that is just the "SDP" syntax described by RFC 4572. As commented in ORTC there is no real reason for that:

w3c/ortc#315

Modernize C

https://matt.sh/howto-c

Server as peer to process decoded video frames

Hey,

This looks like an awesome module. I'm about to dig in deep, but before I do, just want to make sure it can do what I'm hoping:

I'd like my Node server to act as a WebRTC peer and allow me to process decoded video frames in real time on the server.

Is it doable? Any pointers in the right direction? I've started reading through the code in mediasoup/test, but am still not quite sure where I might be able to get access to the video stream.

Thanks!

Implement "Multiple RTP Header Extensions"

• http://tools.ietf.org/html/rfc5285

Set a proper license

People cannot take the risk of looking at your code unless you license it properly.

Revert RTCRtpCodecRtxParameters

In both WebRTC 1.0 and ORTC, RTX "codecs" (RFC 4588) are signaled as common codecs, so we must revert the RTCRtpCodecRtxParameters entry in RtpCodecParameters.

what's the date for 1.0.0?

i am looking for several webrtc server lib in my next project.

mediasoup looks awesome.

BUNDLE and payload values constraint

According to BUNDLE draft, if the same payload value appears more than twice in m= lines belonging to the same BUNDLE group, then those codecs MUST be the very same.

Not yet completely clear what would happen if the same transport is used to receive the same codec (payload, parameters, etc) from different participants (H264 video from Alice, from Bob, from Carol, etc).

• Thread open in MMUSIC WG: http://www.ietf.org/mail-archive/web/mmusic/current/msg16017.html
• Thread open in AVTCORE WG: https://mailarchive.ietf.org/arch/msg/avt/D-zKnht0xKJl6LKWa91FVstKXCw
• Also reported in ORTC: w3c/ortc#407

Should depend on python2

@saghul should this configure.py require python2?:

https://github.com/ibc/mediasoup/blob/master/worker/scripts/configure.py#L1

Implement backpressure in TCP/UDP

libuv offers (now) public API to check the write/send queue size (in bytes). So:

• Add a setter SetBackPressure(value).
• When in Write() or Send() check if such a value is greater than the handle write/send queue.
• If so call listener->onBackPressureExceeded(current_value).

Communicate with another mediasoup ?

Did ice lite implementation has ability that connect with other lite implements?
Or can mediasoup communicate with another one?

Consider using Botan TLS C++ library

• http://botan.randombit.net
• DTLS and DTLS-SRTP: randombit/botan#27

Use srtp_set_user_data() and srtp_get_user_data()

Available in libsrtp since cisco/libsrtp#66.

Use std::unique_ptr instead of std::auto_ptr?

There are some std::auto_ptr in RtpSender, RtpReceiver, etc. According to http://www.cplusplus.com/reference/memory/auto_ptr/:

Automatic Pointer [deprecated]
Note: This class template is deprecated as of C++11. unique_ptr is a new facility with a similar functionality, but with improved security (no fake copy assignments), added features (deleters) and support for arrays. See unique_ptr for additional information.

Upgrade to OpenSSL 1.1.0

There are some API changes to be considered, see https://github.com/cisco/libsrtp/pull/211/files

Re-define RtpParameters IDL

Don't reuse same SSRC (err_status_replay_old)

Info here: cisco/libsrtp#68 (comment)

Usage Example/Getting Started Documentation

What?

Readme needs a usage example for people interested in getting started with the package

Why?

Would love to test drive this package but it's lacking clear explanation of how to get started. Would be great if this could be added to the README or the website.

Note: The api documentation is great but not the best reference for getting started.

Call for Feature Requests

Now that mediasoup is close to its first 1.0.0 release, it's a good chance for users to propose new and cool API features.

Let's do that in here!

Add support in the JS API to send arbitrary RTP packets

Something in the sender side equivalent to the on packet event in the receiver side:

Maybe soemthing like:

RtpSender.send(packet: Buffer|RtpObject)

API to get peers certificate (how to create a Certificate in Node given a PEM)

In 85e698b I've added transport.dtlsRemoteCert which (once DTLS is connected) provides the peer's certificate in PEM format.

Example:

transport.on('dtlsstatechange', (dtlsState) =>
{
    if (dtlsState === 'connected')
        console.warn('remote certificate:\n%s', transport.dtlsRemoteCert);
});

This prints:

remote certificate:
-----BEGIN CERTIFICATE-----
MIIBmTCCAQKgAwIBAgIEAitymDANBgkqhkiG9w0BAQsFADARMQ8wDQYDVQQDDAZX
ZWJSVEMwHhcNMTYwNTEwMTkyMTI4WhcNMTYwNjA5MTkyMTI4WjARMQ8wDQYDVQQD
DAZXZWJSVEMwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANPh1C7jFo3lnWJh
flnueuC1HeSrurt3JHXxK7sYBB1+RNoeCBBPCSHwS51U253SRcCuChupCuZl4E4/
pUz0yzanR0HYQzFrL06KfLSwYoAf40LKKAfjNxSCfSgObRF4LAGs42pGa91U+iHt
Ct4jDGUPLVieDzJBXx3Q/DEHSN8rAgMBAAEwDQYJKoZIhvcNAQELBQADgYEAavSb
CHV+3+l0s+HbVw79bVYPjSBTUAdziaV48QBkxwoNjXmkGlmg59pfhCYytFYWmEHI
5UHX9u2Z5H2UWJ7gTw+Z4fHDZV7BOv8U0Zsz791Mm+1gMhzOV+FATpHLJbRl0GF/
NsSQw9h8tjGfsH3cALPYl88yRGy6GeplPLjcL4M=
-----END CERTIFICATE-----

This can be useful for the app to check the certificate and prove some kind of WebRTC identity mechanism and so on.

But... I need a way to create a crypto.Certificate instance in Node given a cert in PEM format, and I don't find how to do that... :(

Performace comparison

I would like to see a performance comparison to other sfu's

transport.setRemoteParameters() must admit a sequence of fingerprints

Based on w3c/ortc#317, transport.setRemoteParameters() must admit a sequence of remote fingerprints and the worker subprocess must attempt to validate all of them until it finds a valid one (and fail if no one is valid).

it works !!!!!

i am testing the new webrtc api,

let peerconnection = mediasoup.webrtc(room,'alice',{}); 

will just stuck there.

create_parse_bye test fails in Linux

[18:11:21] worker/out/Debug/mediasoup-worker-test
minimum_header .................................................... PASS
buffer_is_too_small ............................................... PASS
version_is_zero ................................................... PASS
length_is_wrong ................................................... PASS
type_is_unknown ................................................... PASS
parse_sdes_chunk .................................................. PASS
create_sdes_chunk ................................................. PASS
parse_sender_report ............................................... PASS
create_sender_report .............................................. PASS
parse_receiver_report ............................................. PASS
create_receiver_report ............................................ PASS
create_parse_bye .................................................. FAIL ***
parse_rtpfb_nack_item ............................................. PASS
create_rtpfb_nack_item ............................................ PASS
parse_rtpfb_tmmb_item ............................................. PASS
parse_rtpfb_tllei_item ............................................ PASS
parse_rtpfb_ecn_item .............................................. PASS
parse_psfb_sli_item ............................................... PASS
parse_psfb_rpsi_item .............................................. PASS
parse_psfb_fir_item ............................................... PASS
parse_psfb_tst_item ............................................... PASS
parse_psfb_vbcm_item .............................................. PASS
parse_psfb_lei_item ............................................... PASS
parse_psfb_afb .................................................... PASS

----------------------------------------------------------------------------

FAILED TESTS


../test/test-rtcp.cpp(339):
    chq_eq_int: 2222 != 2222
../test/test-rtcp.cpp(352):
    chq_eq_int: 2222 != 2222

Check "Codec-Independent Selective Forwarding" draft-aboba-avtcore-sfu-rtp

https://tools.ietf.org/html/draft-aboba-avtcore-sfu-rtp-00

RtpReceiver.js: Add API to receive RTP packets

Add a setter into the RtpReceiver class than activates RTP reception via JS. This would allow:

• Stream recording handled by the JS app.
• Using the RTP somewhere else.

Initial proposal:

let rtpReceiver = peer.RtpReceiver(...);

rtpReceiver.listenForRtp = true;

rtpReceiver.on('rtp', (packet) =>
{
  // packet is a Node Buffer instance
  console.log('RTP packet received');
});

codec + configuration

mediasoup needs to hold a map of payloadType and codecs+configuration. So I need to know what "codec+configuration" means (note that some codecs have their own configuration parameters...).

Mail sent to AVT WG: https://www.ietf.org/mail-archive/web/avt/current/msg17068.html

https://en.wikipedia.org/wiki/RTP_audio_video_profile

Implement more RTCP stuff

mediasoup will not relay RTCP packets but consume and generate them. This is, mediasoup will act as a media endpoint by negotiating and managing RTCP with each client.

• Docs: https://github.com/ibc/mediasoup/wiki/RTCP

Evaluate picojson

A single header file:

• https://github.com/eluck/picojson

Better WebRTC API

• Implement RTCRtpTransceiver object.
• Better handling of Unified-Plan (Firefox) and Plan-B (Chrome).
• Don't require sending audio+video initial SDP offer from the client. Instead:
  • Allow receiving initial SDP offer from client with just audio or video, and just in receive mode.
  • Allow mediasoup to generate the SDP offer for new clients.

NOTE: This may change, and probably will do. More in comments below.

Check new ICE related drafts

• draft-thatcher-ice-renomination: ICE Renomination: Dynamically selecting ICE candidate pairs
• draft-ietf-ice-rfc5245bis: ICE: A Protocol for Network Address Translator (NAT) Traversal
• draft-uberti-mmusic-nombis: Improvements to ICE Candidate Nomination

Add announced IPv4/IPv6 settings

In AWS or Google Cloud, the host has private IP. However we need to announce a public IP address in the ICE candidates generated by mediasoup.

Implement simulcast

mediasoup must be ready to accept simulcast from clients and choose the appropriate simulcast profile to forward to other participants based on their network bandwidth or other parameters (such as selection via API).

• Check RtpStreamSend health to dynamically select a lower simulcast profile (do probation).
• Wait for a RTP full frame before switching the Consumer simulcast profile to another one.
• Rewrite VP8 PictureId and TTL0IDX.
• Be ready for 1 byte PictureId:
  • https://bugs.chromium.org/p/webrtc/issues/detail?id=7877
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1402242

• Docs: https://github.com/ibc/mediasoup/wiki/Simulcast-&-SVC

Add API documentation

Since mediaosup provides a Node ES6 API, so esdoc seems a beautiful solution for auto-generated doc.

I can't install mediasoup on ubuntu 14.04 lts

5901 verbose stack Error: [email protected] postinstall: make Release
5901 verbose stack Exit status 2
5901 verbose stack at EventEmitter. (/root/node-v7.2.1-linux-x64/lib/node_modules/npm/lib/utils/lifecycle.js:255:16)
5901 verbose stack at emitTwo (events.js:106:13)
5901 verbose stack at EventEmitter.emit (events.js:191:7)
5901 verbose stack at ChildProcess. (/root/node-v7.2.1-linux-x64/lib/node_modules/npm/lib/utils/spawn.js:40:14)
5901 verbose stack at emitTwo (events.js:106:13)
5901 verbose stack at ChildProcess.emit (events.js:191:7)
5901 verbose stack at maybeClose (internal/child_process.js:885:16)
5901 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)
5902 verbose pkgid [email protected]

New RTP parameters approach

Let's be in sync with ORTC. This issue clarifies lot of stuff.

Consider RTP matching rules of JSEP

Discussed in:

• w3c/ortc#546 (comment)
• https://mailarchive.ietf.org/arch/msg/rtcweb/YhZlBWCn5yK7-Trr-mcqJIsUDsA

Implement netstrings protocol in the Channel

• https://en.wikipedia.org/wiki/Netstring

C library:

• https://github.com/PeterScott/netstring-c

Node side stuff:

• https://github.com/pgriess/node-netstring/
• https://www.npmjs.com/package/network-byte-order

DTLS handshake fragmentation

https://tools.ietf.org/html/rfc6347#section-4.2.2

Solution in OpenSSL:

#define RTPPAYLOADSIZE 1350

// Disable automatic MTU discovery.
SSL_CTX_set_options(ssl_ctx, SSL_OP_NO_QUERY_MTU);

// Set MTU of datagrams so it fits in an UDP packet.
SSL_set_mtu(ssl, RTPPAYLOADSIZE);
// DTLS_set_link_mtu(ssl, MTU);   // ¿?¿?¿?
BIO_ctrl(write_bio, BIO_CTRL_DGRAM_SET_MTU, RTPPAYLOADSIZE, NULL);

Implement DataChannel

The lib: https://github.com/sctplab/usrsctp

Update to libsrtp2

yep

Got errors when running npm install mediasoup --save

I got errors below when running npm install mediasoup --save

6293 error Linux 4.8.6-x86_64-linode78
6294 error argv "/usr/bin/nodejs" "/usr/bin/npm" "install" "mediasoup" "--save"
6295 error node v7.4.0
6296 error npm v4.0.5
6297 error code ELIFECYCLE
6298 error [email protected] postinstall: make Release
6298 error Exit status 2
6299 error Failed at the [email protected] postinstall script 'make Release'.
6299 error Make sure you have the latest version of node.js and npm installed.
6299 error If you do, this is most likely a problem with the mediasoup package,
6299 error not with npm itself.
6299 error Tell the author that this fails on your system:
6299 error make Release
6299 error You can get information on how to open an issue for this project with:
6299 error npm bugs mediasoup
6299 error Or if that isn't available, you can get their info via:
6299 error npm owner ls mediasoup
6299 error There is likely additional logging output above.

Check multi-stream related drafts

Drafts

• "Plan B: a proposal for signaling multiple media sources in WebRTC": https://tools.ietf.org/html/draft-uberti-rtcweb-plan
• "Sending Multiple Types of Media in a Single RTP Session": https://tools.ietf.org/html/draft-ietf-avtcore-multi-media-rtp-session
• "Sending Multiple RTP Streams in a Single RTP Session": https://tools.ietf.org/html/draft-ietf-avtcore-rtp-multi-stream
• "Sending Multiple RTP Streams in a Single RTP Session: Grouping RTCP Statistics and Other Feedback": https://tools.ietf.org/html/draft-ietf-avtcore-rtp-multi-stream-optimisation

Others

• https://lists.w3.org/Archives/Public/public-ortc/2016Apr/0020.html

can not find a way to make webrtc api work with mediasoup api

mediasoup api is designed for ortc, it seems that the signalingServer only need to handle candidates.

webrtc still need to offer/answer sdp info， can you give me an example how to make webrtc and mediasoup work?

use case about mediasoup

hi，

mediasoup is a greate lib, and i can't wait to have a try.

my use case is that at least two publisher and tens of viewer in one room, can mediasoup be used like this?

Need a better Makefile and built-in dependencies

I don't want that MediaSoup depends on 3rd libraries globally installed (openssl, libsrtp, etc). Let's imitate the Node way (all the dependencies included in the source tree). And then use gyp or ninja or lalala.

/cc @saghul

Implements getStats()

Identifiers for WebRTC's Statistics API

Being these two elements the ones with most important for now:

• RTCInboundRTPStreamStats
• RTCOutboundRTPStreamStats

Can I connect mediasoup with browser

I need browser client A push mediastream to mediasoup ,then mediasoup send the mediasteam to browser client B,but I can't connect mediasoup with browser. Broswer use ice-full ,but mediasoup use ice-lite,so,I don't know how to use ice-lite in brower.