Comments (11)
two things to keep in mind about RTWebSocket when looking at the above links:
- RTWebSocket uses the same primary abstraction as RTMFP: unidirectional ordered message-oriented "flows" named by arbitrary binary metadata, where each flow can have an independent priority/precedence (that you can change at any time), each message can have an arbitrary transmission deadline (that you can change at any time), and "return flow association" generalizes bidirectional communication into arbitrarily-complex trees of unidirectional flows
- to fully realize the "Real-Time" benefits of RTWebSocket (or RTMFP), you need to avail yourself of the real-time facilities of priority/precedence and transmission deadlines.
note that you can revise a message's transmission deadline after it's been queued (for example, you can revise the transmission deadlines of previously-queued video messages when you get a new keyframe).
the link to Section 5.3.5 of RFC 7425 above is to an illustration of multiple levels and many more than 2 flows of a bidirectional flow tree.
RFC 7016 describes RTMFP (a UDP-based transport protocol suspiciously similar to, but significantly predating, Quic). RFC 7425 describes how to send RTMP video, audio, data, and RPC messages over RTMFP. the same method can be used to send RTMP messages over RTWebSocket, and the https://github.com/zenomt/rtwebsocket and https://github.com/zenomt/rtmfp-cpp repos have demonstrations of doing that.
from selkies-gstreamer.
I am starting to believe that for both WebTransport and WebRTC, using custom Python solutions outside of GStreamer such as https://github.com/aiortc/aioquic or https://github.com/aiortc/aiortc is becoming plausible, since as long as an RTP stream is generated with GStreamer, WebRTC/WebTransport operations can definitely be separated.
This can be done soon instead of waiting for GStreamer and with much less funding.
This is because using GStreamer plugins for transport is slowly showing less flexibility and performance issues, because they are mainly focused on media processing rather than protocols, and browser issues are fixed much slower. Using a dedicated library for WebRTC or WebTransport QUIC will solve transport specific issues more versatilely.
While the core developers ourselves are short of time to do this in the meantime, and we will focus maintaining the GStreamer native webrtcbin/webrtcsink pipelines, people are welcome to set bounties or help out in this way instead of implementing a GStreamer plugin. A substantial bounty may allow us to implement this ourselves.
This will likely solve #7 for once and for all.
from selkies-gstreamer.
https://www.w3.org/TR/webcodecs-hevc-codec-registration/
HEVC is slowly coming.
from selkies-gstreamer.
Dependent on:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1571
from selkies-gstreamer.
https://www.phoronix.com/news/Mozilla-Firefox-114
Firefox now supports WebTransport as of 114. WebGPU is also recently implemented and prototyped as of 116.
Firefox still requires the implementation of the WebCodecs API, but that makes Safari the only major browser engine without WebTransport.
from selkies-gstreamer.
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1261#note_2053233
A hypothesis I have is that Issue #7 might indeed be something from the web browser client.
If true, we need our own audio stack going through MSE using RTCDataChannel or an equivalent WebTransport interface, it would be sensible to use external WebRTC libraries as well as a new signalling server. Since the only thing we can use from GStreamer's webrtcbin
would be the video sink.
Solved differently in #7. Left for historical information.
from selkies-gstreamer.
#7 is fixed. But WebTransport support in web browsers made things very hard for serving in localhost or for self-signed certificates. Good technology but needs waiting.
from selkies-gstreamer.
There are restricted environments where only Connection: Upgrade
to WebSocket from plain HTTP is available (such as banks).
This might require aiohttp as well as python-websockets.
It might be a good idea to pair WebSocket + WebCodecs, perhaps instead of the widely intended WebTransport + WebCodecs pairing.
WebCodecs is better than MSE or WASM in terms of latency, and it can be used to be paired with WebTransport as well. That way, both WebSocket and WebTransport will use the same media decoder if we implement both.
In conclusion, I think implementing good old WebSockets has its point.
from selkies-gstreamer.
Things that are relevant for using WebRTC without exposing various ports for the TURN server, perhaps.
(Continuously updated)
https://github.com/amitv87/turn_ws_proxy
https://groups.google.com/g/turn-server-project-rfc5766-turn-server/c/un2u2xsrjLA
from selkies-gstreamer.
Archive of relevant discussions from #7
Opus + MP4? https://chromestatus.com/feature/5100845653819392
- https://github.com/parsec-cloud/parsec-sdk/blob/master/sdk/parsec.h
- https://github.com/evshiron/parsec-web-client
- https://github.com/robterrell/web-client
https://www.w3.org/2018/12/games-workshop/slides/21-webtransport-webcodecs.pdf
Looks like WebTransport + WebCodecs (in implementation within Firefox) is the modern alternative to WebAssembly or MSE. In progress to be implemented in Firefox.
I’ve been tracking WebTransport closely and prototyping it locally off and on the past year.
It’s interesting but not ready for the Selkies Project yet. Right now, it works well in a tightly controlled production environment. However, today it requires a lot of configuration, valid certificates, browser command line arguments and OS exceptions that make working it locally very challenging.
Self-signed certificates look possible but complicated. By the way, is mainly using WebSocket that bad (for a fall-back protocol) compared to WebRTC? - Since it's what Guacamole, NoVNC, etc uses.
https://docs.libp2p.io/concepts/transports/webtransport/
When connecting to a WebSocket server, browsers require the server to present a TLS certificate signed by a trusted CA (certificate authority). Few nodes have such a certificate, which is the reason that WebSocket never saw widespread adoption in the libp2p network. libp2p WebTransport offers a browser API that includes a way to accept the server’s certificate by checking the (SHA-256) hash of the certificate (using the serverCertificateHashes option), even if the certificate is “just” a self-signed certificate. This allows us to connect any browser node to any server node, as long as the browser knows the certificate hash in advance (see WebTransport in libp2p for how WebTransport addresses achieve this).
WebTransport in libp2p
WebTransport multiaddresses are composed of a QUIC multiaddr, followed by /webtransport and a list of multihashes of the node certificates that the server uses.
For instance, for multiaddress /ip4/192.0.2.0/udp/123/quic/webtransport/certhash/, a standard local QUIC connection is defined up until and including /quic. Then, /webtransport/ runs over QUIC. The self-signed certificate hash that the server will use to verify the connection.
The WebTransport CONNECT request is sent to an HTTPS endpoint. libp2p WebTransport server use /.well-known/libp2p-webtransport. For instance, the WebTransport URL of a WebTransport server advertising /ip4/192.0.2.0/udp/1234/quic/webtransport/ would be https://192.0.2.0:1234/.well-known/libp2p-webtransport?type=noise (the ?type=noise refers to the authentication scheme using Noise).
Yeah, i can second what @danisla is saying... local development for webtransport is hectic at the moment especially for anything working on with the browser (server-side quic development is the easiest).
For example, for warp we must generate self-signed certs every 14 days, as well as a fingerprint hash as a workaround to having to execute bash scripts each time we need to refresh our site in order to force Chrome to allow QUIC connections.
Also, there is currently no way to load balance QUIC connections, UDP load balancers are great and all but they take away the QUIC's ability for roaming support (you can change internet connections, e.g mobile data to Wifi, without having to reinitiate the handshake to the server)
QUIC is shiny, promising and all... however we are not there yet
from selkies-gstreamer.
I have been informed by @zenomt that RTWebSocket is a pretty interesting option for the WebSocket portion of this project.
https://github.com/zenomt/rtwebsocket
https://github.com/zenomt/rtmfp-cpp
https://www.rfc-editor.org/rfc/rfc7016.html
Return flow association (bi-directional information):
https://www.rfc-editor.org/rfc/rfc7016.html#section-2.3.11.1.2
https://www.rfc-editor.org/rfc/rfc7425.html
Return flow association (bi-directional information):
https://www.rfc-editor.org/rfc/rfc7425.html#section-5.3.5
Along with WebTransport, the possible reason to use this is to show frames as soon as they arrive instead of going through internal jitterbuffers which WebRTC has limited control over (but the current approach of having separate audio and video streams should still make the frame display as fast as possible).
from selkies-gstreamer.
Related Issues (20)
- Support remote controlling retro console devices HOT 10
- Support Selkies Joystick Interposer in 32-bit (especially Wine), support manual installation in addition to DEB HOT 2
- Long hold ESC to exit full screen in the web interface does not work in certain cases and many key combinations leak to the client OS HOT 10
- [META] Rewrite gst-web HTML5 interface frontend HOT 5
- Audio randomly gets disabled after latest changes to webrtcbin HOT 12
- Various Resize/DPI improvements with the user interface and the cursor HOT 7
- Support extended stylus or touch gestures HOT 1
- x264 encoder failing to initialize HOT 3
- Add option to manually specify XAuthority files for login greeters HOT 3
- Audio does not play on "START" button press HOT 5
- Cannot start stream with NvFBC HOT 4
- Implement WebRTC statistics logging in host HOT 2
- Keyboard keystrokes continue to be held when packets are lost by congestion or unstable connection / Improve the last bit of keyboard latency overall while accommodating unstable connections HOT 3
- Instead of erroring with no audio server, restore server-side --disable_audio CLI option or automatically detect no PulseAudio server
- Achieve higher efficiency and quality in low-bandwidth and high-latency environments
- Improve Opus: reduce NetEQ jitterbuffer & eliminate discontinuous audio artifacts from jitter with Opus + RED HOT 1
- [META] Optimize the WebRTC stack to the maximum HOT 2
- When menu is open, CPU thread congestion from Vue causes choppy video and frame drop every second HOT 3
- Color is slightly incorrect from the GStreamer video converter
- Divide Dockerfile to various containers in selkies-vdi HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from selkies-gstreamer.