kixelated / moq-rs Goto Github PK
View Code? Open in Web Editor NEWRust library for Media over QUIC
License: Apache License 2.0
Rust library for Media over QUIC
License: Apache License 2.0
Subprotocol Negotiation offers a way to denote that we're running MoQ over WebTransport (similar to what we can do with ALPN in the case of raw QUIC).
I propose we adopt the same string here as for an ALPN value (the current draft says 'moq-00
').
We can add support for this on the server side now, but it won't be useful until the browser WebTransport constructor has a way to signal this, too.
Relates to: kixelated/moq-js#94
We need to host a compelling demo, which means servers running in multiple regions. A user could ingest to a server in America and another user could egress from a server in Europe. I'm thinking we use a global database to keep track of active broadcasts and the origin. We can go directly to the origin instead of a cache tree for now.
I want to use something like terraform to keep any cloud configuration in code. It would likely go in a separate repo because it's specific to the quic.video demo.
The cloud provider doesn't matter too much but I would like it to be sponsored. I have a lot of experience with AWS but also want to use this opportunity to try something new. I'm thinking we start with GCP because there's a better chance we can get free credits, but I'm sure we'll have to port based on whoever is willing to sponsor. We might also be able to get individual hosts from some CDN company
The current library uses a bespoke JSON API. MoQ Transport is finally getting adopted and we should switch to it.
It doesn't support all of the needed features so we can make extensions, but I would like to be backwards compatible. If that's not possible then we should bring it up with the working group.
The draft is quite flexible with how groups/objects can be used. However it makes decoding an absolute pain, so I want to be explicit on how "Warp" runs on top of MoQ Transport. This will be a long running debate within the WG.
Each independent group of pictures is sent over a dedicated QUIC stream. Which means that: CMAF segment == QUIC stream
To map this to MoQ Transport, it means a single object per group with an unspecified size (0). That's how the server works today albeit with a JSON header instead of an object header.
This doesn't need to go to everyone, just the publisher who erred.
It's annoying to run mkcert every 14 days. We should do it automatically on process start for local development.
Then I can transmux QUIC media stream into rtmp/hls/http-flv formats in my project https://github.com/harlanc/xiu
The relays form a mesh network. This won’t scale past a few nodes.
If you use moq-pub
with a duplicate broadcast, you'll get an error:
Error: failed to create MoQ Transport session
Caused by:
0: decode error: io error: connection lost
1: io error: connection lost
2: connection lost
3: closed by peer: 91141958511331
A few issues here:
91141958511331 = 409
or somethingduplicate
.Documented known issue.
WIP to address in #60
We should experiment with transmitting objects over datagrams.
I don't think a cache for datagrams makes any sense. These are fire-and-forget.
Develop a GStreamer plugin.
Enable Continuous Deployment for Fly.io relay server by generating a deploy token and wiring up GitHub Actions.
https://fly.io/docs/app-guides/continuous-deployment-with-github-actions/
(We may transition over to the new Google Cloud hosts soon, but it'd be nice to keep the current relay automatically in sync with main
until we're fully ready to cut over)
The client should be able to choose which tracks to SUBSCRIBE to. However, we should also implement a SUBSCRIBE_AUTO message that lets the server choose the current track based on network conditions.
The current transport API uses unidirectional streams only. The MoQ Transport (#27) adds a bidirectional stream for the SETUP message, but then it uses an unspecified number of unidirectional streams going forward.
We should just use a single bidirectional streams for all control messages (moq-wg/moq-transport#138). It makes the implementation dramatically simpler, avoids handshake race conditions, and is similar to how HTTP/3 works. We should do it despite the draft lagging.
The current media source reads until the end of the file, where it presumably exits.
To loop the media source, we need to restart from the beginning of the file and rewrite any timestamps. Alternatively, we could create new tracks, and somehow signal a discontinuity, but I don't think that's viable or a good idea. The ffmpeg fragmentation command should already remove any byte offsets so those shouldn't be a problem.
Note that the MP4 crate can produce atoms, but it doesn't work very well. The problem is that the decode step throws out any unknown atoms/fields so they're lost on the re-encode. I would personally modify the raw bytes directly; it should be as simple as replacing a uint32.
Make it possible to use moq-rs from languages other than Rust.
This would be part of supporting direct integration with ffmpeg as an output protocol.
I added the current UTC time using ffmpeg and displayed on the browser.
Unfortunately, clock sync is a bitch.
We should have a better way to measure latency, potentially something like NTP inside of the MoqTransport protocol itself. It gets complicated as multiple relays are introduced so I'm welcome to any ideas.
I'm working on a proof-of-concept that aims to demonstrate video streaming from one machine to another using QUIC.
My first goal in this project is to demonstrate streaming "via localhost" - i.e. running the relay and publisher on the same machine as I view the stream on. To do this have been following the Local Development guide. The machine in question is running Windows 11, and the moq-rs binaries are all running inside a WSL2 Ubuntu 20.04 container.
Errors I am seeing
I am running into problems when trying to view the stream in a browser. This is what I see in various browsers I have tried:
dom.media.webcodecs.enabled
flag set to true
results in the same error as above.The steps I am following are:
dev/source.mp4
../dev/cert
to generate the certificate../dev/relay
to start the relay. It compiled without errors initially, and now runs without needing to compile anything. When I run this, I see the relay start up and it says it is listening on port 4443.relay
running), run ./dev/pub
. It appears to run correctly, and the bottom line of output is updated once per second and shows an increasing frame counter.https://quic.video/watch/Hefring?server=localhost:4443
. Hefring is the name of my machine, which happens to be set in the $NAME env var. At this point I see the errors described above.Other things I have tried
Adding the WSL2 network adapter's address to the cert
command, so the cert includes it in the call to mkcert
, just like it does for 127.0.0.1
. No change, the same errors are shown.
Running the same setup on my home server, which runs dedicated Ubuntu 20.04. However, on this machine, the pub
script does not successfully manage to start ffmpeg
(no frames are sent). However, running pub-file
does produce an output file on disk!
Questions
mkcert
tool just fine (I think).Any help with this would be greatly appreciated. I'm not really sure what's going on here - I'm a bit of a newbie to this sort of thing! And if you are able to share the details of a known working setup, perhaps I can use that to help diagnose my issue.
I have attached logs from a run of the relay and publisher, in case they help:
moq-rs-logs-1.txt
Thanks for any help in advance!
In current moq-pub version, a MOQ group is created for each GOP in a video. For low latency, this is preferable. However, tuning the number of GOPs in a group would be nice.
Add a module that uses ffmpeg to generate and publish media. This would be the precursor for something like OBS support.
The quinn BBR code has a lot of WARNING text. I enabled it anyway because bufferbloat means the death of any prioritization and real-time ambitions.
Somebody should validate that BBR is working as expected. I would use qlog/qvis to monitor the congestion metrics and eyeball if everything looks good for a few crappy sessions. I can tell you from experience that it's very easy to screw up pacing and application-limited flows.
Also, we may want to disable the PROBE_RTT phase when application-limited. I think this would have a lot of impact for live media, even more than switching over to BBRv2.
The current server can only use a disk source. We need to add the ability to receive a live stream and redistribute it.
This should be relatively straight forward because of the reusable media producer objects. There should be a trait Source
so we can switch between disk/net sources on a whim.
Clients establish a QUIC connection per broadcast. We should investigate reusing connections via ANNOUNCE like the specification intends.
The current server starts pushing media the instant a WebTransport connection is established. This is easy, but it's both a waste of bandwidth and a blocker for ABR (#25).
We should add a SUBSCRIBE message so the client can ask for individual tracks. This is part of the MoQ Transport draft already but hasn't been implemented because I'm lazy.
I keep getting 'Opening handshake failed' in brave v1.46.144 (running on chromium v108.0.5359.128) and chrome canary v111.0.5494.0
I have tried searching all over the internet but nothing seems to solve this.
This includes:
Chrome /Chromium seems to have dropped support for the flag --ignore-certificate-errors-spki-list
Run warp in the browser without TLS_CERTIFICATE error
I am a newbie to media domain. I am curious about switching this project from golang to rust and why c/c++ was not considered.
Everything is in terraform but otherwise everything is manual, including the number of instances.
Develop an ffmpeg plugin. Piping to stdin currently works but it could be improved.
I'm experiencing choppy audio with persistent background noise, and I've tried everything to eliminate it. The only solution I've found so far is to modify the value at https://github.com/wanjohiryan/warp-1/blob/43af19ba94dae9c9ea72f50164c2128c79a39c02/player/src/audio/index.ts#LL18C13-L18C61
js ring: new RingInit(2, 4410), // 100ms at 44.1khz
to
js ring: new RingInit(2, 4410 * 5),` // 500ms at 44.1khz
Do you have any suggestions on how to fix this issue?"
We should either (or both):
The WebTransport library we're using doesn't expose stream priorities, even though the underlying QUIC library supports it.
I'm thinking we fork the library and add a field. In the long-run, I'm not use the h3::webtransport
crate is even the best fit, because it has a HTTP/3 centric API. I would advocate for a minimal layer on top of quinn that ONLY supports WebTransport and implements the bare minimum HTTP/3 internals behind the scenes.
The current code is intended to run on a server. However, we should add client support, we can stream media to a remote server. This is a prerequisite to running as an OBS plugin.
The biggest problem is that the WebTransport library we're using only supports servers.
The current disk source only exposes a single rendition. This restricts the congestion control response to only skipping the end of a segment.
We should encode multiple renditions with ffmpeg. We should continue to use a single MP4 file as input instead of separate files.
There’s no logging, metrics, or even health checks. I don’t even know if the site is up.
When I directly run make run
on my machine, it prompts to let me input password, but still failed due to permission denied. It seems to be working fine when I run sudo make run
. Should the README mention using this command as root?
OS: Debian 12
Full error message:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 704, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 399, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.11/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/lib/python3.11/http/client.py", line 975, in send
self.connect()
File "/usr/lib/python3/dist-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 788, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/six.py", line 718, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 704, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 399, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.11/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/lib/python3.11/http/client.py", line 975, in send
self.connect()
File "/usr/lib/python3/dist-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 547, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 33, in <module>
sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 81, in main
command_func()
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 200, in perform_command
project = project_from_options('.', options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 60, in project_from_options
return get_project(
^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 152, in get_project
client = get_client(
^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/compose/cli/docker_client.py", line 41, in get_client
client = docker_client(
^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/compose/cli/docker_client.py", line 170, in docker_client
client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 197, in __init__
self._version = self._retrieve_server_version()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Pe
rmission denied'))
make: *** [Makefile:5: run] Error 1
Develop an OBS plugin.
Quinn lacks the tracing required to validate the user experience. We should add qlog and other useful metrics.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.