livepeer / catalyst Goto Github PK

Minimally we'll want to auto-populate the retry-join flag so we can get rid of the two separate config files in the config directory. It can easily be set with the CATALYST_RETRY_JOIN environment variable once #51 ships.

Some other cases that we'd want once we can generate configs:

• Testing with transcoding enabled and transcoding disabled
• Testing with RTMPS enabled/disabled on the load balancer side

Starting with the Docker images

https://discord.com/channels/423160867534929930/992554008776540180/1006636908836827267

having issues with 4K streams. At some point I thought they just didn't work, but now only transcoding is broken. only source renditions is available

likely transcoding is taking too long and then being ignored as if it timed out? i know it's normally close to/worse than real-time for 4K videos

live for the next ~2h:
https://lvpr.tv/?url=https://sao-canary-catalyst-0.livepeer.fun/hls/video+196a6qz7pg06nl5h/index.m3u8?mute=false

Streaming command:

ffmpeg -re -i ./vintage/vintage-cercle.mp4 -c:v copy -c:a copy -strict -2 -f flv rtmp://sao-canary-catalyst-0.livepeer.
fun/live/196a-em8j-4822-1hun

File is 4K with H264 and AAC as required. Can send it over to someone but I don't think there's anything special with it. Works fine on livepeer.com.

Discovered while writing #57: we're feeding IP addresses rather than hostnames into the Mist load balancer:

▶ curl 'http://localhost:8042/video+PLAYBACK_ID'
185.102.219.180

It would be good to use hostnames instead so that we can redirect people to valid HTTPS URLs and do DTSCS properly.

Check out this build. It's for commit cc9d188, right? Here's manifest.yaml for that commit, showing MistServer version DDVTECH/mistserver@2c33311.

However, the build itself downloads DDVTECH/mistserver@52e1cb9. This shows up in the logs when it does verification:

#24 120.5 I0815 18:17:07.185310     474 downloader.go:67] Doing SHA checksum verification for service=mistserver
#24 120.7 I0815 18:17:07.375632     474 shasum.go:13] Verifying shasum with file=52e1cb9c35555ca67ab215d3f80919b344fb0c32_checksums.txt

I'm guessing the offending error is this, occurring right at the git clone step?

fatal: Not a valid object name cc9d1881159057f782222e725d03a8f3dd047634^{commit}

There's a start here but we need to account for the DTSCS routing (should be another IngressRouteTCP using SNI routing to route incoming requests to the right place) and the externally-routable HTTPS requests.

We also want to move to separate Kubernetes namespaces for everything, so that should be integrated alongside livepeer/livepeer-infra#875

Following the pattern set by go-livepeer:

livepeer-darwin-amd64.tar.gz
livepeer-linux-amd64.tar.gz
livepeer-windows-amd64.zip

Particularly trying to detect failures shown in #65

Probably this'll mean just allowing Mist to pass through stuff that's in a format it's not expecting?

We need to handle incoming requests for e.g. https://playback.livepeer.studio/hls/PLAYBACK_ID/index.m3u8 and 302 redirect them to the address of a Catalyst server that has the stream in question. Previously, we proxied all of these requests behind the scenes, but that led to some suboptimal performance characteristics, especially around segment download latencies.

The workflow will be basically:

1. Handle GET https://playback.livepeer.studio/hls/PLAYBACK_ID/index.m3u8
2. Turn around and ask MistUtilLoad for the best place to watch the pertinent stream, like so:

▶ curl 'http://localhost:8042/video+PLAYBACK_ID'
185.102.219.180

3. 302 Redirect the user to that URL https://185.102.219.180/hls/PLAYBACK_ID/index.m3u8
4. If no stream is present, 404

There will be some nuance in getting this right, I think... for example, we'll want to check for the X-Forwarded-Proto: https header and return an HTTPS url if present, that sort of thing.

@iameli TODO: Document why we're doing the flow this way

After make-dev in latest catalyst repo:

Error: unknown shorthand flag: 'j' in -j
Usage:
  livepeer-transcode [command]

Available Commands:
  help         Help about any command
  list-presets Lists transcoding presets
  transcode    Transcodes video file using Livepeer API

Flags:
  -a, --api-host string   Livepeer API host (default "livepeer.com")
  -k, --api-key string    Livepeer API key
  -h, --help              help for livepeer-transcode
  -v, --version           version for livepeer-transcode

Use "livepeer-transcode [command] --help" for more information about a command.

@leszko Please add a comment about the stream redirection logic

@emranemran Please add a comment about DTSC(S) stream replication

@hjpotter92 Please add a comment explaining the the e2e test flow in this repo

https://www.notion.so/livepeer/Multi-Node-Stream-Replication-Docs-43f2d8982da042f8894ad452f99cf9fc

Blocked by https://github.com/livepeer/livepeer-infra/issues/917

Canary branch works like #55, except:

• Instead of scraping for the latest semver releases, we scrape the the bucket releases for each project's main branch.
• We don't need a PR for canary like we have with staging, because we expect upstream projects to push new semver releases when they're ready to go

Initial Setup:

• Get rid of docker-compose stuff
• Rename Mist config to ~/.config/livepeer/catalyst.json
• Document the above
• Remove mist-api-connector
• Remove authWebhookUrl from broadcaster
• Add a default stream named "video"
• Add a default transcoding configuration to "video" with a hardcoded_broadcasters field set to [{"address":"http://localhost:8935"}]
• Document streaming into that video+foo field using ffmpeg and OBS (gstreamer too!)
• Document playing back that stream at http://localhost:8080/hls/video+foo/index.m3u8
• Add catalyst-node
• Document playing back from catalyst-node at http://localhost:8090/hls/foo/index.m3u8

Build-to-run workflow:

• make mistserver for Mist
• For other projects, e.g. make livepeer, make livepeer-catalyst-node
• Once you've recompiled something, rather than running make dev again, you can run e.g. killall livepeer and let MistController bring it back up

• Upon an upstream project cutting a new semver release (make release or whatever), a new CI action gets kicked off
• This CI action generates a new "version manifest" file based on the latest semver releases of all of the upstream projects listed in manifest.yaml
• This version manifest then gets committed onto a new branch off of main named staging. This branch is always force-pushed, so there's just one commit to merge when we want to cut a release.
• It should also open a PR, if one does not already exist, from staging to main.
• When we want to cut a new Catalyst release, we simply merge that PR and CI takes care of publishing new images.

There is some race condition, which makes the execution of the livepeer/catalyst container flaky. In result, sometimes, when catalyst is started, there are no logs at all and the only running process is MistController.

Steps to reproduce

Start multiple livepeer catalyst and find the ones with no logs

for i in $(seq 0 100); do CID=$(docker run -d --rm livepeer/catalyst); sleep 5; LOGS=$(docker logs $CID); [[ -z "$LOGS" ]] || docker kill $CID;done

This command will start 100 container one by one and kill all expect the ones that produce no logs. Then check the running containers and if there are none, then retry the command above. In my case, there was always at least one running container left.

$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED         STATUS         PORTS                                    NAMES
79e3ec38e0c6   livepeer/catalyst              "/usr/bin/MistContro…"   8 minutes ago   Up 8 minutes   1935/tcp, 4242/tcp, 8080/tcp, 8889/udp   sleepy_poincare

2. Exec into the container and check the running processes

You should find that the only running process is MistController.

$ docker exec -it 79e3ec38e0c6 sh
# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  16080  4340 ?        Ssl  13:47   0:00 /usr/bin/MistController -c /etc/livepeer/catalyst.json
root        14  0.1  0.0   2060   496 pts/0    Ss   13:57   0:00 sh
root        20  0.0  0.0   5480  2384 pts/0    R+   13:57   0:00 ps aux

Initial targets: Linux AMD64, Linux ARM64, macOS AMD64, macOS ARM64

• Queries all pertinent repositories using GitHub API, downloads latest release following project-platform-arch.tar.gz format, untars to bin

Blocked by DDVTECH/mistserver#61; blocking auto-deploys of Mist

A lot of our testing is going to be Docker integration testing. That makes it important to have fast rebuilds... and currently, we have an ADD . . command that means everything gets redownloaded on every change.

Consolidating all of the downloader stuff in cmd/downloader should fix that; it probably won't share code with other things like catalyst-node anyway.

We need have a suite of integration tests and run them as part of the CI pipeline.

This GH Issue includes:

• Create an integration test to test the DMS stream replication
• Include the test in the test GH Action workflow

Test scenario:

1. Start 3 DMS instances
2. Send stream to a single instance
3. Read playback from each instance and check that it's correct

At first glance, it seems we could use test-containers and start a few DMS instances locally. We already used test-containers library inside go-livepeer, for the purpose of E2E Testing.

Currently stream in Livepeer Studio can have two prefixes, corresponding to two Mist stream configurations: video for normal streams, and videorec for streams with recording enabled.

So our playback URLs look like this:

https://catalyst-node.example.com/hls/video+PLAYBACK_ID/index.m3u8
https://catalyst-node.example.com/hls/videorec+PLAYBACK_ID/index.m3u8

But in the dashboard, our playback URLs don't have any prefixes, and look like this:

https://catalyst-node.example.com/hls/PLAYBACK_ID/index.m3u8

The legacy system uses etcd routes and middleware in Traefik to handle this sort of thing, but we're getting rid of that. Other Mist configurations use the defaultStream parameter to automatically rewrite PLAYBACK_ID to e.g. stream+PLAYBACK_ID, but that won't work for us because we have two stream configurations, video and videorec.

I'm leaning toward handling this on the catalyst-node side, so it can issue a redirect from https://global-playback-url.example.com/hls/PLAYBACK_ID/index.m3u8 to https://catalyst-node.example.com/hls/PLAYBACK_ID/index.m3u8.

New command line option: -redirectPrefixes=video,videorec. Then when we get an incoming playback request for PLAYBACK_ID, we'll query the Mist load balancer concurrently for video+PLAYBACK_ID and videorec+PLAYBACK_ID. (If an incoming request comes in for video+playbackId we can just forward it as-is.) Then we can issue a redirect to whichever one has a stream, if any.

From time to time, the stream is not load balanced correctly between Mist nodes. I attach the logs of both catalyst nodes.

logs-node-1.txt
logs-node-2.txt

Steps to reproduce

1. Check out the code from the branch with additional debug logs

Check out the branch reproduce-load-balancing-issue. It adds additional logs and increases the timeout of the E2E Test

2. Run many tests sequentially to reproduce the flakiness

for i in $(seq 0 30); do go test -timeout 30m -v ./test/e2e/*; done

3. Notice that a test was stuck

You should see a message that the test is stuck.

#####
STUCK in requireReplicatedStream

4. Check the container logs

Now, you can check the logs of the containers and investigate the issue.

docker ps

There are lots right now. Let's move to like...

1. Primary Catalyst branch
2. Mac-compatible branch

In #86, we added stream interruption test i.e. verify playback still recovers after ingest stream is killed and restarted. However, we set it to forcefully pass even with failures - this was temporary until we root-caused the issue. With #65 now resolved, this stream interruption test can be correctly set to fail if playback is not able to recover correctly during an ingest stream interruption.

Commit is here: https://github.com/DDVTECH/DMS/pull/105

So we need that to work and then we need to teach MistController/MistUtilLoad to utilize it when connecting across to different regions.

mistserver builds in catalyst is broken and reports the following when running make mistserver:

Builing release "Generic_64" for version "mist-v3.0-130-g3f81819" @ debug level 4
Building with SRT
Building without RIST support
SRT-LIB-IS-XXX="/home/parallels/catalyst/build/compiled/lib/libsrt.a"
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
Not cross compiling - building sourcery
-- Configuring done
-- Generating done
-- Build files have been written to: /home/parallels/mistserver
+ make -j2
make[1]: Entering directory '/home/parallels/catalyst/build/mistserver'
make[1]: *** No targets specified and no makefile found.  Stop.
make[1]: Leaving directory '/home/parallels/catalyst/build/mistserver'
make: *** [Makefile:60: mistserver] Error 2

• Setup a webhook for the project
• Webhook receives information about new branch/commit pushed to go-livepeer/mistserver/etc dependents
• Generate a dynamic manifest file (in gcp bucket?)
• Allow downloader to use manifest file URL in addition to local paths
• Extend downloader to override specific project metadata in manifest (on demand)

It was evident from #54 that we need a third container that launches all tester binaries instead of relying on the CI host to have certain dependencies fulfilled. A workaround was used for the linked PR to copy over the relevant binaries required for load-testing into the CI host. These binaries should instead live in their own containers.

Minimally we need /cmaf/ alongside /hls/, but ideally we'd have a few more... here's a full list taken from the Mist embed page:

http://localhost:8080/stream%2bfoo.sdp
ws://localhost:8080/stream%2bfoo.mp4
http://localhost:8080/hls/stream%2bfoo/index.m3u8
http://localhost:8080/stream%2bfoo.mp4
http://localhost:8080/stream%2bfoo.webm
http://localhost:8081/hls/stream%2bfoo/index.m3u8
http://localhost:8080/cmaf/stream%2bfoo/Manifest
http://localhost:8080/cmaf/stream%2bfoo/index.m3u8
http://localhost:8080/cmaf/stream%2bfoo/index.mpd
rtmp://localhost/play/stream%2bfoo
http://localhost:8080/dynamic/stream%2bfoo/manifest.f4m
http://localhost:8080/stream%2bfoo.flv
rtsp://localhost:5554/stream%2bfoo
http://localhost:8080/stream%2bfoo.ts
http://localhost:8080/stream%2bfoo.aac

Okay, we don't want quite all of those...