mconf / bbb-webrtc-sfu

Control server for WebRTC SFU

bbb-webrtc-sfu

Server for controlling a WebRTC selective forwarding unit (SFU) media server.

bbb-webrtc-sfu's Issues

Improve logging metadata

Review multi-line logs.
Add metadata to logs associating them to fields of interest for postprocessing.

• Some fields of interest:
  • userId
  • meetingId

[mcs-core] Review all event emitters and listeners to check for leaks

We should do a good review of our usage of EventEmitters on mcs-core because there are some leaks hidden in there related to those.

Content sharing doesn't work immediately after taking someone else's presenter during a screensharing

Endpoint(s) description :
WebRTC (BBB Web user) and RTP User

Media format :
Video H.264

bbb-webrtc-sfu version/hash :
current dev branch: ebbe9dd

Problem description / Steps to reproduce:

1. Web User Join through BBB (with moderator access)
2. RTP user join
3. RTP User shares it's content and automatically become a presenter in BBB
4. Web User becomes a presenter (RTP User automatically stops
5. Web User starts screen sharing and no video is shown for Web/RTP User

Expected Behavior:
Video should be shown to all users (Web and RTP) in the room

Extra Info:
After step (5), if user stops screen sharing and start it again, the screen media is sent correctly

[mcs-core] Implement requestKeyframe method and keyframeNeeded event

Of great importance to SFU cases where we have mixed models of keyframe requisition (rtcp fir/nack/pli and SIP INFO, for instance).
One new API method, one new API event:

• requestKeyframe: mediaId => response; Notifies a media that it needs to generate a keyframe somehow
• keyframeNeeded: e(mediaId); Notifies a media that a keyframe was requested from it from someone. Generally triggered by requestKeyframe or by the controller (internally).

Add a configuration option for setting Redis auth credentials

[video][screenshare] Recording stop event might not fire under some circumstances

Reported by @fcecagno.

I was not able to reproduce that yet, but logs are attached. Further investigation is needed.
Slated for a possible patch release on v2.3.x (prolly v2.3.4).

6b9ebdf77061c9b8246179308475adda04c239cb202c0b8797fdc752e7975651-json.log

[mcs-core] Create a buffer for media state events

Create a buffer of limited size to store media state events for media streams. Append the buffer info to info objects traded by the API, mainly when reacting to a new onEvent subscription.

The idea behind this is to give an idea of the current and previous media states for 3rd-party applications which probe media streams from mcs-core.

[mcs-core] Extend BaseProcess class in mcs-core's top level script

Should allow us to re-use some very stable code and prevent the mcs-core process from hanging on some rare occasions.

[mcs-core] Add adapter info to getMediaInfo object

[mcs-core] Implement a captioning system with API calls and events for MCU sessions

Implement log files expiry

Including a configuration option for the expiry period (log.fileExpiry).

[video|screenshare] Hook BBB recording events to recording media flowing event

This should improve the video sync a bit in recordings by actually generating a timestamp once the media flow event is fired instead of right after the recording media is created and brokered through MCS.

[mcs-core] Add timestamps to API responses and events

[mcs-core] Increase number of retries on first-time media server connection attemps

We should have a fixed amount of retries (suggestion: 5) for a first-time connection attempt to a configured media server in the balancer module.
Currently, it'll drop the media server if it can't connect on the first try. This is problematic for setups where startup order isn't guaranteed.

[mcs-core] Add time information for media streams on their info objects

Time information to be added:

• Creation
• Duration
  ...

[sfu-upper-layer] Turn the connection-manager family into a separate process

We should turn the connection-manager family (HttpServer, WebsocketConnectionManager et. al) into a separate process. It's currently tied to the master process and this is bad because it can potentially take down the whole app.

[mcs-core] Refactor the Kurento adapter's stop method

It's currently too big and quite messy because all the transposers cleaning and pipeline accounting/cleaning is inside there. Consider stripping the method into multiple ones. Also look at the stopRecording wart.

[mcs-core] Renegotiated medias are not being indexed correctly

Regression from #80.

Screenshare extra media with incompatible type

The second media can't be connected to others, but it has "sendrecv" as video type.

[mcs-core] Implement the video switching by voice activity strategy

Implement video switching by voice activity.

Includes:

• An N-media working video switching algorithm

Depends on:

• #20
• #29

[mcs-core] Act as offeree in negotiation

• Allow for pre-starting medias with no descriptor offer
• Allow transparent renegotiation of media sessions and medias

Base work done on https://github.com/mconf/bbb-webrtc-sfu/tree/v2.3.2-no-desc-negotiation.
Aimed for v2.4.x

Allow layout and conference floor to be changed by sending DTMF codes

Fix MCU video receiving

[mcs-core] Media unit with bundled audio+video is being messed up with role = offerer

sdp-session's renegotiateStreams is flaky with media units that have audio+video bundled: the remote descriptor is being overwritten with a partial video descriptor, causing the mediaType to be wrongly redefined and messing the negotiation flow.

Consider moving the inactive reappend to the kurento adapter so we don't have to reassing remote descriptors at every step (and consequently fix this).

Only one video from endpoint is composed

Calling redacted server with Multipresença with 2 or more cameras active, only the first one is shown in the compostion.

[mcs-core] subscribedTo prop setting is rendering false positives due to MediaFactory not indexing media units correctly

[mcs-core] Fix content handling in the voice-switching strategy handler

The content handling in there is currently outdated and doesn't work at all.

[mcs-core] Add option to filter candidates that don't match host's IP in Kurento adapter

Add optional filterHostCandidates boolean parameter to Kurento object configuration.
Add optional ipv6 parameter to Kurento object configuration.
Filter outbound candidates that aren't host and that doesn't match the ip and ipv6 configuration parameters when iceLite option is enabled.
Triage with load testing.

[mcs-core] Add option to make subscriber's media specs slave to source media

[video] Allow per-stream bitrate configuration

Slated for 2.3.x, will be uplifted to 2.4.x, 2.5.x.

[mcs-core] Implement the setStrategy API method

Implement the setStrategy/getStrategies API methods to define generic switching algorithms. Includes:

• Per-room and per-user handling
• User and media model updates to track strategies
• An strategy factory and a base strategy class with a pre-defined interface
• Implementation of the optional parameter multiSession on user join or media brokering methods
• Implementation of a base strategy: freewill, which basically means "release from the current strategy" and from there do nothing

[mcs-core] Medias and MEDIA_CONNECTED events are getting duplicated on tracking when acting as offerer

This is due to a quirk when merging 2.3.x to 2.4.x.

[screenshare] Exception when external content is closed (unpublish)

[mcs-core] Refactor the adapter negotiate calls to avoid using Media-* classes inside of them

This is a code debt of mine. We should drop the use of those objects and define a proper JSON response format for the adapter negotiate calls and instantiate the Media-* classes outside of the adapters.

[mcs-core] Implement subscribedTo event

Useful for notifying clients that the source media for a given subscriber media has changed.

[mcs-core] Re-do the `connect`/`disconnect` API methods implementation

The current implementation is botched, too complex and bug-prone in many ways due to the mediaProfile subspec that serves for nothing other than complicating things.
The discrepant behaviour of those methods between the upper *MediaSession models and *Media models is also due for a thorough review.
We should base ourselves around only the connectionType subspec and let go of the content subdivision I introduced (who knows what for).

There's base work done on https://github.com/mconf/bbb-webrtc-sfu/tree/v2.3.2-no-desc-negotiation in the MediaSession model and although that is a badly-optimized prototyped, it's already much better.

Slated for v2.4.x as I'd like to clean that mess as fast as possible.

[mcs-core] Implement getInfo API method

Returns info on the member represented by the identifier parameter:

• getInfo : identifier = undefined -> info:
  • roomId -> room's getInfo;
  • mediaId -> media/media-session's getMediaInfo;
  • userId -> user's getInfo;
  • no parameter -> server capabilities: implemented methods and available adapters.

[mcs-core] Implement mediaRenegotiated event

Sends the media info object when it's renegotiated for those listening to it.

[mcs-core] Implement a layout system with API calls and events for MCU sessions

[mcs-core] Extend the voice-switching-handler algorithm to handle MCU sessions

[mcs-core] Implement proper parameterization for bandwidth/bitrate/REMB in the Kurento adapter

There was a stub for those implemented on bd12f2a, but the values are hardcoded.
Consider implementing a way to parameterize those on media creation according to the configured or specified, via API, media specs.
That's a tad cumbersome today because we'd either need to fetch those values from the SDP (hence needing to add another parse-and-fetch utilitary method on sdp-wrapper for that) OR devise a proper data format and flow between the model/factory classes and the media server adapter classes.

Slated for v2.5.x. Might get this done earlier on v2.4.x if anybody picks this soon enough.

[mcs-core] Implement a BaseMediaServerAdapter to properly define adapter interfaces

Add option to specify media types to be generated on offerer in pub/sub/pubsub calls

Add the medias optional parameter which is an object/hash map of the following format:

medias: {
   audio: String|Array[String]|undefined ->  "sendrecv"|"recvonly"|"sendonly",
   video: String|Array[String]|undefined ->  "sendrecv"|"recvonly"|"sendonly",
   content: String|Array[String]|undefined ->  "sendrecv"|"recvonly"|"sendonly",
}

When the key values are arrays of the specified possible strings, it'll behave as a quantity specifier.

[mcs-core] Add some sort of reconnection procedure to new ESL wrapper

Needs to be able to reconnect to ESL and re-fetch stuff when FreeSWITCH goes off, is restarted or the TCP socket drops.
Right now the wrapper only throws a fit with some uncaught exceptions and doesn't reconnect. No good.

Endpoint receives multiple equal compositions

Calling redacted server with Multipresença (endpoint with 6 receive media lines) it receives multiple compositions repeated.

[mcs-core] Implement mediaType-based media server balancing

Implemented in https://github.com/mconf/bbb-webrtc-sfu/tree/v2.3.8-dev-balancer-mediatypes.

TODO:

• Fix Firefox DTLS negotiation bug
• Merge with 2.4.0
• Open PR to 2.4.0-dev

[mcs-core] Finish implementing the mDNS candidate resolution in the Kurento/FreeSWITCH adapters

There is a stub implemented for that on 758e47f which just parsed and ignored the candidates.

We need to find a proper way/lib to resolve mDNS addresses. Chrome started shipping this spawn of satan too (only mobile Safari did that before), so we need to resolve this ASAP.

Slated for v2.4.0

Add remainder of implementation for DTMF tones

Slated for 2.3.10 and will be uplifted 2.4.x, 2.5.x.
dtmfSent, dtmf and dtmfReceived handling.

[mcs-core] Balancer stream accounting is buggy and rendering negative numbers

v2.4.0 issue. Stream count decrements are flaky, I'm seeing negative numbers. It doesn't break anything, but might prevent proper transposing.

No video in the composition using Avermedia Capture Card with static image and Multipresença

Using Multipresença to call redacted server with camera Avermedia U3 sometimes the video is not shown in the composition (all black), but it is OK in the web.
It seems related to keyframes, tried to reduce gop in the encoder and then it works.