codec + configuration about mediasoup

[ibc]

Response by Magnus:

10.1.2. Payload Type (PT) Value Reuse

Multiple bundled "m=" lines might represent RTP based media. As all
RTP based media associated with a BUNDLE group belong to the same RTP
session, in order for a given payload type value to be used inside
more than one bundled "m=" line, all codecs associated with the
payload type number MUST share an identical codec configuration.
This means that the codecs MUST share the same media type, encoding
name, clock rate and any parameter that can affect the codec
configuration and packetization.
[I-D.ietf-mmusic-sdp-mux-attributes] lists SDP attributes, whose
attribute values must be identical for all codecs that use the same
payload type value.

So identical a=rtpmap and a=fmtp lines would meet this criteria.

However, two different m= blocks configuring a particular PT for a codec could possibly use different values for an optional parameter that can be received correctly by either configurations.

Codecs that can produce different encoding bit-rates can in many cases be used at different bit-rates and fulfil this requirement.

The underlying issue is that what is identical codec configurations are codec specific. Thus hard to write a precise definition of it.

[ibc]

So let's check how Chrome Canary M51 and Firefox Nightly 48 offer "codecs":

• Chrome:

m=audio 49606 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 126
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10; useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:126 telephone-event/8000
a=maxptime:60

m=video 58734 UDP/TLS/RTP/SAVPF 100 101 116 117 96 97 98
a=rtpmap:100 VP8/90000
a=rtpmap:101 VP9/90000
a=rtpmap:116 red/90000
a=rtpmap:117 ulpfec/90000
a=rtpmap:96 rtx/90000
a=fmtp:96 apt=100
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=101
a=rtpmap:98 rtx/90000
a=fmtp:98 apt=116

• Firefox:

m=audio 64761 UDP/TLS/RTP/SAVPF 109 9 0 8
a=rtpmap:109 opus/48000/2
a=fmtp:109 maxplaybackrate=48000;stereo=1
a=rtpmap:9 G722/8000/1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000

m=video 53852 UDP/TLS/RTP/SAVPF 120 126 97
a=rtpmap:120 VP8/90000
a=fmtp:120 max-fs=12288;max-fr=60
a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1

[ibc]

As I commented in the AVT ML:

Offer by Firefox 48:

m=audio 49706 UDP/TLS/RTP/SAVPF 109 9 0 8
a=rtpmap:109 opus/48000/2
a=fmtp:109 maxplaybackrate=48000;stereo=1
a=rtpmap:9 G722/8000/1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000

m=video 61083 UDP/TLS/RTP/SAVPF 120 126 97
a=rtpmap:120 VP8/90000
a=fmtp:120 max-fs=12288;max-fr=60
a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1

Answer by Chrome 51:

m=audio 61998 UDP/TLS/RTP/SAVPF 109 9 0 8
a=rtpmap:109 opus/48000/2
a=fmtp:109 minptime=10; useinbandfec=1
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000

m=video 9 UDP/TLS/RTP/SAVPF 120
a=rtpmap:120 VP8/90000

1a) opus in Firefox:

a=rtpmap:109 opus/48000/2
a=fmtp:109 maxplaybackrate=48000;stereo=1

1b) opus in Chrome:

a=rtpmap:109 opus/48000/2
a=fmtp:109 minptime=10; useinbandfec=1

So I must assume that maxplaybackrate, stereo, minptime and useinbandfec are optional params for opus, or that their default values (if omitted) match those values, so both codecs+configuration are the "same".

2a) VP8 in Firefox:

a=rtpmap:120 VP8/90000
a=fmtp:120 max-fs=12288;max-fr=60

2b) VP8 in Chrome:

a=rtpmap:120 VP8/90000

Here also I assume that max-fs and max-fr are optional parameters that do not prevent both offered codec+configuration to match.

3a) G722 in Firefox:

a=rtpmap:9 G722/8000/1

3b ) G722 in Chrome:

a=rtpmap:9 G722/8000

Do they match? Is "1" the default value for the 'encoding' field in a rtpmap entry?

4) H264 in Firefox:

a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1

a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1

Since Firefox offers two H264 codecs I must assume they are, indeed, different codecs. So it seems that "packetization-mode=1" make them different (which also means that its default value is not 1). Right?

This is terribly complex and unmanageable. And this requires the SFU developer to become an expert in codec specific parameters (regardless the SFU won't decode media).

[ibc]

Regarding H264 packetization-mode parameter:

https://tools.ietf.org/html/rfc6184#section-6

6.2. Single NAL Unit Mode

This mode is in use when the value of the OPTIONAL packetization-mode
media type parameter is equal to 0 or the packetization-mode is not
present. All receivers MUST support this mode.

So valid values are 0 (default if not set), 1 and 2. Mandatory for WebRTC are 0 and 1.

[ibc]

Chrome 50 with H264 support (enabled in chrome://flags):

a=rtpmap:107 H264/90000

Just that. No fmtp line. Since no packetization-mode is given then it gets its default value 0. What about profile-level-id?

/cc @saghul

[saghul]

If the attribute is absent the default is used: 420010.

[ibc]

Wow, so indeed Firefox and Chrome do not interop at H264 level...

[ibc]

Moving comments form #18 here:

UPDATE: After a good chat with @jmillan, the room will be provided with mandatory videoCodec and audioCodec parameters so endpoints MUST support them and MUST use them (and no other codecs). Period.

Still we need an internal list of "valid" audio/video codecs in mediasoup. The list (to be completed) would start with:

• Audio:
  • opus (default)
  • alaw
  • ulaw
• Video:
  • VP8 (default)
  • H264 packetization-mode=0
  • H264 packetization-mode=1

@saghul said:

If 2 different endpoints use different H264 profile-level-id attributes someone will get confused, I'm afraid.

The profile-level-id can also make H264 codecs different. A low powered device might not be able to handle the "high" profile, for example.
On the implementations I've worked with, we always accept whatever the remote wants, but I guess this need not be the case for eveyone.

@ibc said:

If so, different profile-level-id values would also generate different codec units. May you please detail those values?

@saghul said:

They could. There are different profiles and levels. Also check https://tools.ietf.org/html/rfc6184#section-8.1

@ibc said:

For example, as shown in #19, Firefox offers two H264 configurations:

a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1

Does it mean that it does not support other profile-level-id?

@saghul said:

No. Those 2 configurations use the same profile-level-id, but different packetization-mode, hence the need for 2 configs.

[saghul]

Wow, so indeed Firefox and Chrome do not interop at H264 level...

Not necessarily. One of them could just accept the offer and use the supplied profile-level-id. Here they both use the same profile (42, baseline) and different level (1f == 31 == 3.1 vs 10 == 1.0). Different levels could cause problems, but more importantly, different profiles could (42 == baseline vs 4d == main), for example.

[ibc]

Interesting.... so what? XD

[ibc]

At the end I need specific and separate "codec configurations" to be thread as the "same codec+configuration".

Note however that codec parameters given by each peer will be properly indicated to each receiver peer (so Chrome will be the H264 params of Firefox and vice-versa), but still I need a proper list of "codec + configurations".

[notedit]

no vp9 codecs？

[ibc]

mediasoup is a SFU so it does not encode/decode codecs. This means that any codec will be supported.

[ollyde]

I have an issue where https://v3demo.mediasoup.org/?roomId=aaaahh works

But using the default codecs as explained here fails locally, does anyone know what config the demo server is running?

Fails for Web (Chrome)

{
	kind      : 'audio',
	mimeType  : 'audio/opus',
	clockRate : 48000,
	channels  : 2
},
{
	kind       : 'video',
	mimeType   : 'video/VP8',
	clockRate  : 90000,
	parameters :
	{
		'x-google-start-bitrate' : 1000
	}
},