kixelated / moq-rs Goto Github PK

We should either (or both):

• Allow users to specify a particular namespace as a command line argument
• Generate UUIDs as part of the namespaces (or maybe just as the default if nothing is specified) to avoid conflict

I am a newbie to media domain. I am curious about switching this project from golang to rust and why c/c++ was not considered.

Add a module that uses ffmpeg to generate and publish media. This would be the precursor for something like OBS support.

Documented known issue.

WIP to address in #60

Subprotocol Negotiation offers a way to denote that we're running MoQ over WebTransport (similar to what we can do with ALPN in the case of raw QUIC).

I propose we adopt the same string here as for an ALPN value (the current draft says 'moq-00').

We can add support for this on the server side now, but it won't be useful until the browser WebTransport constructor has a way to signal this, too.

Relates to: kixelated/moq-js#94

Clients establish a QUIC connection per broadcast. We should investigate reusing connections via ANNOUNCE like the specification intends.

Error:

I keep getting 'Opening handshake failed' in brave v1.46.144 (running on chromium v108.0.5359.128) and chrome canary v111.0.5494.0

I have tried searching all over the internet but nothing seems to solve this.

This includes:

1. Generating new certificates,
2. allowing insecureOrigins in tls.Config
3. Using a different machine

Probable cause

Chrome /Chromium seems to have dropped support for the flag --ignore-certificate-errors-spki-list

Expected behaviour

Run warp in the browser without TLS_CERTIFICATE error

Screenshot

In current moq-pub version, a MOQ group is created for each GOP in a video. For low latency, this is preferable. However, tuning the number of GOPs in a group would be nice.

I added the current UTC time using ffmpeg and displayed on the browser.

Unfortunately, clock sync is a bitch.

We should have a better way to measure latency, potentially something like NTP inside of the MoqTransport protocol itself. It gets complicated as multiple relays are introduced so I'm welcome to any ideas.

The relays form a mesh network. This won’t scale past a few nodes.

The current transport API uses unidirectional streams only. The MoQ Transport (#27) adds a bidirectional stream for the SETUP message, but then it uses an unspecified number of unidirectional streams going forward.

We should just use a single bidirectional streams for all control messages (moq-wg/moq-transport#138). It makes the implementation dramatically simpler, avoids handshake race conditions, and is similar to how HTTP/3 works. We should do it despite the draft lagging.

The current server starts pushing media the instant a WebTransport connection is established. This is easy, but it's both a waste of bandwidth and a blocker for ABR (#25).

We should add a SUBSCRIBE message so the client can ask for individual tracks. This is part of the MoQ Transport draft already but hasn't been implemented because I'm lazy.

Enable Continuous Deployment for Fly.io relay server by generating a deploy token and wiring up GitHub Actions.

https://fly.io/docs/app-guides/continuous-deployment-with-github-actions/

(We may transition over to the new Google Cloud hosts soon, but it'd be nice to keep the current relay automatically in sync with main until we're fully ready to cut over)

The quinn BBR code has a lot of WARNING text. I enabled it anyway because bufferbloat means the death of any prioritization and real-time ambitions.

Somebody should validate that BBR is working as expected. I would use qlog/qvis to monitor the congestion metrics and eyeball if everything looks good for a few crappy sessions. I can tell you from experience that it's very easy to screw up pacing and application-limited flows.

Also, we may want to disable the PROBE_RTT phase when application-limited. I think this would have a lot of impact for live media, even more than switching over to BBRv2.

Develop a GStreamer plugin.

When I directly run make run on my machine, it prompts to let me input password, but still failed due to permission denied. It seems to be working fine when I run sudo make run. Should the README mention using this command as root?

OS: Debian 12

Full error message:

Traceback (most recent call last):                                                                                     
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 704, in urlopen                                
    httplib_response = self._make_request(                                                                             
                       ^^^^^^^^^^^^^^^^^^^                                                                             
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 399, in _make_request                          
    conn.request(method, url, **httplib_request_kw)                                                                    
  File "/usr/lib/python3.11/http/client.py", line 1282, in request                                                     
    self._send_request(method, url, body, headers, encode_chunked)                                                     
  File "/usr/lib/python3.11/http/client.py", line 1328, in _send_request                                               
    self.endheaders(body, encode_chunked=encode_chunked)                                                               
  File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders                                                  
    self._send_output(message_body, encode_chunked=encode_chunked)                                                     
  File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output                                                
    self.send(msg)                                                                                                     
  File "/usr/lib/python3.11/http/client.py", line 975, in send                                                         
    self.connect()                                                                                                     
  File "/usr/lib/python3/dist-packages/docker/transport/unixconn.py", line 30, in connect                              
    sock.connect(self.unix_socket)                                                                                     
PermissionError: [Errno 13] Permission denied                                                                          
                                                                                                                       
During handling of the above exception, another exception occurred:                                                    
                                                                                                                       
Traceback (most recent call last):                                                                                     
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 489, in send                                        
    resp = conn.urlopen(                                                                                               
           ^^^^^^^^^^^^^                                                                                               
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 788, in urlopen                                
    retries = retries.increment(                                                                                       
              ^^^^^^^^^^^^^^^^^^                                                                                       
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 550, in increment                                  
    raise six.reraise(type(error), error, _stacktrace)                                                                 
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                 
  File "/usr/lib/python3/dist-packages/six.py", line 718, in reraise                                                   
    raise value.with_traceback(tb)                                                                                     
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 704, in urlopen                                
    httplib_response = self._make_request(                                                                             
                       ^^^^^^^^^^^^^^^^^^^                                                                             
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 399, in _make_request                          
    conn.request(method, url, **httplib_request_kw)                                                                    
  File "/usr/lib/python3.11/http/client.py", line 1282, in request                                                     
    self._send_request(method, url, body, headers, encode_chunked)                                                     
  File "/usr/lib/python3.11/http/client.py", line 1328, in _send_request                                               
    self.endheaders(body, encode_chunked=encode_chunked)                                                               
  File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders                                                  
    self._send_output(message_body, encode_chunked=encode_chunked)                                                     
  File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output                                                
    self.send(msg)                                                                                                     
  File "/usr/lib/python3.11/http/client.py", line 975, in send                                                         
    self.connect()
  File "/usr/lib/python3/dist-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 33, in <module>
    sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 200, in perform_command
    project = project_from_options('.', options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 60, in project_from_options
    return get_project(
           ^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 152, in get_project
    client = get_client(
             ^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/cli/docker_client.py", line 41, in get_client
    client = docker_client(
             ^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/cli/docker_client.py", line 170, in docker_client
    client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Pe
rmission denied'))
make: *** [Makefile:5: run] Error 1

The current disk source only exposes a single rendition. This restricts the congestion control response to only skipping the end of a segment.

We should encode multiple renditions with ffmpeg. We should continue to use a single MP4 file as input instead of separate files.

We need to host a compelling demo, which means servers running in multiple regions. A user could ingest to a server in America and another user could egress from a server in Europe. I'm thinking we use a global database to keep track of active broadcasts and the origin. We can go directly to the origin instead of a cache tree for now.

I want to use something like terraform to keep any cloud configuration in code. It would likely go in a separate repo because it's specific to the quic.video demo.

The cloud provider doesn't matter too much but I would like it to be sponsored. I have a lot of experience with AWS but also want to use this opportunity to try something new. I'm thinking we start with GCP because there's a better chance we can get free credits, but I'm sure we'll have to port based on whoever is willing to sponsor. We might also be able to get individual hosts from some CDN company

I'm experiencing choppy audio with persistent background noise, and I've tried everything to eliminate it. The only solution I've found so far is to modify the value at https://github.com/wanjohiryan/warp-1/blob/43af19ba94dae9c9ea72f50164c2128c79a39c02/player/src/audio/index.ts#LL18C13-L18C61
js ring: new RingInit(2, 4410), // 100ms at 44.1khz
to
js ring: new RingInit(2, 4410 * 5),` // 500ms at 44.1khz
Do you have any suggestions on how to fix this issue?"

The current code is intended to run on a server. However, we should add client support, we can stream media to a remote server. This is a prerequisite to running as an OBS plugin.

The biggest problem is that the WebTransport library we're using only supports servers.

The client should be able to choose which tracks to SUBSCRIBE to. However, we should also implement a SUBSCRIBE_AUTO message that lets the server choose the current track based on network conditions.

I'm working on a proof-of-concept that aims to demonstrate video streaming from one machine to another using QUIC.

My first goal in this project is to demonstrate streaming "via localhost" - i.e. running the relay and publisher on the same machine as I view the stream on. To do this have been following the Local Development guide. The machine in question is running Windows 11, and the moq-rs binaries are all running inside a WSL2 Ubuntu 20.04 container.

Errors I am seeing
I am running into problems when trying to view the stream in a browser. This is what I see in various browsers I have tried:

• Firefox prod v119.0.1 results in the error "NotSupportedError: WebTransport constructor: No support for serverCertificateHashes yet". This might be expected, as the Issues pages does mention no Firefox support for WebCodecs yet - however this is not the sort of error message I would expect in this case!
• Firefox nightly v121.0a1, with the dom.media.webcodecs.enabled flag set to true results in the same error as above.
• Chrome prod v119.0.6045.124 results in the error "WebTransportError: Opening handshake failed."

The steps I am following are:

• Installed go via apt and cargo via the install helper script they provide.
• Installed ffmpeg v4.4.4-0ubuntu1~20.04.sav1.1
• Downloaded the sample video to dev/source.mp4.
• Run ./dev/cert to generate the certificate.
• Run ./dev/relay to start the relay. It compiled without errors initially, and now runs without needing to compile anything. When I run this, I see the relay start up and it says it is listening on port 4443.
• In a separate terminal (leaving relay running), run ./dev/pub. It appears to run correctly, and the bottom line of output is updated once per second and shows an increasing frame counter.
• I then try and view the stream by visiting the URL the publisher prints out: https://quic.video/watch/Hefring?server=localhost:4443. Hefring is the name of my machine, which happens to be set in the $NAME env var. At this point I see the errors described above.

Other things I have tried

• Adding the WSL2 network adapter's address to the cert command, so the cert includes it in the call to mkcert, just like it does for 127.0.0.1. No change, the same errors are shown.

• Running the same setup on my home server, which runs dedicated Ubuntu 20.04. However, on this machine, the pub script does not successfully manage to start ffmpeg (no frames are sent). However, running pub-file does produce an output file on disk!

Questions

• Have I understood the basic setup correctly? Should what I am doing (run the relay in the background, then run the publisher, then view the stream) work out of the box?
• It looks to me to be a certificate issue - but I'm not sure why, as I was able to run the customised mkcert tool just fine (I think).
• What browser (and version) is known to work to stream data from the relay?

Any help with this would be greatly appreciated. I'm not really sure what's going on here - I'm a bit of a newbie to this sort of thing! And if you are able to share the details of a known working setup, perhaps I can use that to help diagnose my issue.

I have attached logs from a run of the relay and publisher, in case they help:
moq-rs-logs-1.txt

Thanks for any help in advance!

Make it possible to use moq-rs from languages other than Rust.

This would be part of supporting direct integration with ffmpeg as an output protocol.

If you use moq-pub with a duplicate broadcast, you'll get an error:

Error: failed to create MoQ Transport session

Caused by:
    0: decode error: io error: connection lost
    1: io error: connection lost
    2: connection lost
    3: closed by peer: 91141958511331

A few issues here:

1. The error code is not being converted from the WebTransport number space back to the application number space. 91141958511331 = 409 or something
2. We should have a 1:1 mapping between error codes so we can display the human-readable text. In this case, duplicate.
3. Or errors have an optional "reason" so we should display that instead.

It's annoying to run mkcert every 14 days. We should do it automatically on process start for local development.

We should experiment with transmitting objects over datagrams.

1. We need to add datagram support to webtransport-quinn.
2. moq-relay should listen for incoming datagrams.
3. moq-relay should transmit outgoing datagrams to matching subscriptions.

I don't think a cache for datagrams makes any sense. These are fire-and-forget.

There’s no logging, metrics, or even health checks. I don’t even know if the site is up.

The current server can only use a disk source. We need to add the ability to receive a live stream and redistribute it.

This should be relatively straight forward because of the reusable media producer objects. There should be a trait Source so we can switch between disk/net sources on a whim.

The current library uses a bespoke JSON API. MoQ Transport is finally getting adopted and we should switch to it.

It doesn't support all of the needed features so we can make extensions, but I would like to be backwards compatible. If that's not possible then we should bring it up with the working group.

The draft is quite flexible with how groups/objects can be used. However it makes decoding an absolute pain, so I want to be explicit on how "Warp" runs on top of MoQ Transport. This will be a long running debate within the WG.

Each independent group of pictures is sent over a dedicated QUIC stream. Which means that: CMAF segment == QUIC stream

To map this to MoQ Transport, it means a single object per group with an unspecified size (0). That's how the server works today albeit with a JSON header instead of an object header.

Develop an ffmpeg plugin. Piping to stdin currently works but it could be improved.

Develop an OBS plugin.

The current media source reads until the end of the file, where it presumably exits.

To loop the media source, we need to restart from the beginning of the file and rewrite any timestamps. Alternatively, we could create new tracks, and somehow signal a discontinuity, but I don't think that's viable or a good idea. The ffmpeg fragmentation command should already remove any byte offsets so those shouldn't be a problem.

Note that the MP4 crate can produce atoms, but it doesn't work very well. The problem is that the decode step throws out any unknown atoms/fields so they're lost on the re-encode. I would personally modify the raw bytes directly; it should be as simple as replacing a uint32.

The WebTransport library we're using doesn't expose stream priorities, even though the underlying QUIC library supports it.

I'm thinking we fork the library and add a field. In the long-run, I'm not use the h3::webtransport crate is even the best fit, because it has a HTTP/3 centric API. I would advocate for a minimal layer on top of quinn that ONLY supports WebTransport and implements the bare minimum HTTP/3 internals behind the scenes.

Everything is in terraform but otherwise everything is manual, including the number of instances.

This doesn't need to go to everyone, just the publisher who erred.

Then I can transmux QUIC media stream into rtmp/hls/http-flv formats in my project https://github.com/harlanc/xiu

Quinn lacks the tracing required to validate the user experience. We should add qlog and other useful metrics.