occasional failed replication or high (~15s) cold start time about catalyst

[emranemran]

@Thulinma, here's a log where it failed on first try (timeout after ~15s) and then worked on a subsequent attempt. Can you please take a look?

---

Steps to repro:

• ingest into lax using this command:

ffmpeg -re -f lavfi -i testsrc=size=480x320:rate=30 -f lavfi -i sine -vf drawtext="text='%{localtime\:%X}':fontsize=20:fontcolor=white:x=7:y=7" -pix_fmt yuv420p -vcodec libx264 -vb 2000k -preset ultrafast -acodec aac -f flv 'rtmp://lax-canary-catalyst-0.livepeer.fun/live/4ce7-030b-5ty3-lmbb’

• Use https://hls-js.netlify.app/demo and attempt streaming from lon url:

https://lon-canary-catalyst-0.livepeer.fun/hls/video+4ce74xm3c29sx2jw/index.m3u8

• Initially got a manifestLoadTimeout after a few seconds. It lines up with the following in the attached logs:

2022-08-19 11:38:43	[2022-08-19 18:38:43] MistInDTSC:video+4ce74xm3c29sx2jw (7156) MEDIUM: Track 1 resizing (reason: not enough parts): frags 60->60, keys 60->60, parts 1800->3600, pages 10->10
2022-08-19 11:38:47	[2022-08-19 18:38:47] MistController (8) MEDIUM: Local API access automatically authorized
2022-08-19 11:38:52	[2022-08-19 18:38:52] MistController (8) MEDIUM: Local API access automatically authorized
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: Would select tracks: 0, 1, 2, 3, 4, 5 (6)
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: Adding 6 tracks
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: Would select tracks: 0, 1, 2, 3, 4, 5 (6)
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: Adding 6 tracks
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: initialize
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: initialize
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: initialize
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: Would select tracks: 0, 1, 2, 3, 4, 5 (6)
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: Adding 6 tracks
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: Client handler shutting down, exit reason: connection closed
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) INFO: Executing CONN_CLOSE trigger: http://localhost:7933/ (asynchronous)
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: Posting to http://localhost:7933/ (1/5)
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistOutHLS:video+4ce74xm3c29sx2jw (7151) MEDIUM: disconnect
2022-08-19 11:38:53	[2022-08-19 18:38:53] MistController (8) MEDIUM: UDP API: {"trigger_stat":{"ms":1,"name":"CONN_CLOSE","ok":true}}rue}}3dbc3744441af2664d40":"UA:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"}}

• Subsequent attempt to playback worked and lines up here in the logs:

2022-08-19 11:39:02	[2022-08-19 18:39:02] MistOutHTTP (7167) INFO: Received request: /hls/video 4ce74xm3c29sx2jw/index.m3u8 => HLS (video+4ce74xm3c29sx2jw)
2022-08-19 11:39:02	[2022-08-19 18:39:02] MistOutHTTP (7167) MEDIUM: Switching from HTTP () to HLS (video+4ce74xm3c29sx2jw)
2022-08-19 11:39:02	[2022-08-19 18:39:02] MistOutHTTP (7167) MEDIUM: disconnect
2022-08-19 11:39:02	[2022-08-19 18:39:02] MistOutHTTP (7167) MEDIUM: Matching: 0.0.0.0 to 84.17.50.99 with 97 prefix
2022-08-19 11:39:02	[2022-08-19 18:39:02] MistOutHLS:video+4ce74xm3c29sx2jw (7167) MEDIUM: Received request: /hls/video 4ce74xm3c29sx2jw/index.m3u8 => HLS (video+4ce74xm3c29sx2jw)
2022-08-19 11:39:02	[2022-08-19 18:39:02] MistOutHLS:video+4ce74xm3c29sx2jw (7167) MEDIUM: initialize
2022-08-19 11:39:02	[2022-08-19 18:39:02] MistOutHLS:video+4ce74xm3c29sx2jw (7167) MEDIUM: Stream video+4ce74xm3c29sx2jw already active; continuing

---

A couple of observations after tracing through the code:

• In the first attempt, Mist receives the playback request and then spews a lot of this for 5s or so:

MistInDTSC:video+4ce74xm3c29sx2jw (7156) INFO: Unhandled command: ok

Looking at input_dtsc.cpp, this would indicate there's some unknown command in the packet stream for the dtsc input stream - is that expected?

• We also see a lot of this after MistOutHLS disconnects after first playback attempt - are we still processing the input dtsc stream here? Also, not quite sure what this is referring to after reviewing the code:

2022-08-19 11:38:56	[2022-08-19 18:38:56] MistInDTSC:video+4ce74xm3c29sx2jw (7156) INFO: Key duration 2000 is quite long - confirming with upstream source
2022-08-19 11:38:56	[2022-08-19 18:38:56] MistInDTSC:video+4ce74xm3c29sx2jw (7156) INFO: Key duration 2000 is quite long - confirming with upstream source
2022-08-19 11:38:56	[2022-08-19 18:38:56] MistInDTSC:video+4ce74xm3c29sx2jw (7156) INFO: Key duration 2000 is quite long - confirming with upstream source
2022-08-19 11:38:56	[2022-08-19 18:38:56] MistInDTSC:video+4ce74xm3c29sx2jw (7156) INFO: Key duration 2000 is quite long - confirming with upstream source

lon-stream-fail-then-pass.log

[Thulinma]

Looking at input_dtsc.cpp, this would indicate there's some unknown command in the packet stream for the dtsc input stream - is that expected?

Yes. The other side is simply acknowledging with "ok" that the key durations are okay (see below).

We also see a lot of this after MistOutHLS disconnects after first playback attempt - are we still processing the input dtsc stream here?

If by "processing" you mean receiving, then yes. There is no "processing" as such; data is simply added to the media buffers as it arrives. The whole point of DTSC is to remove the need for any kind of fancy processing. ^_^

Also, not quite sure what this is referring to after reviewing the code:

If there are GOPs with a duration more than twice of other GOPs, Mist becomes suspicious of stream integrity and checks with the other end if that's okay or something is wrong. If this is okay, nothing further happens. If it does not match upstream GOP durations, the stream is disconnected to clear up any possibly integrity loss.

My guess is that part of the problem is an unstable keyframe interval. Try adding -g 30 to the ffmpeg command and see if that helps?
It also would appear from the logs that the upstream Mist instance may have a fairly large DVR window ( = buffer size) configured. Try setting it to e.g. 20000 ( = 20s ) and see if that helps any?
I tried to stream using the provided keys and URLs, but was not able to see the feed. Should I do something specific to enable something somewhere?
I can also take a closer look at what's happening if I can get shell access to the two Mist nodes involved; there's tons of commands that allow inspecting stream buffer state etc.

[Thulinma]

Oh, and on a side note: why are you using a random hls.js instance to test, instead of the built-in player here? https://lon-canary-catalyst-0.livepeer.fun/video+4ce74xm3c29sx2jw.html?dev=1

[yondonfu]

I tried to stream using the provided keys and URLs, but was not able to see the feed. Should I do something specific to enable something somewhere?

This guide https://www.notion.so/livepeer/Multi-node-Catalyst-Guide-f9d11a291cfa4fdeb3f849c73f969ce0 should have the required information. If there is no playback then something is likely not working properly.

[yondonfu]

I also just observed a really long replication time to LON when streaming into NYC.

The command for streaming into NYC:

ffmpeg -re -f lavfi -i testsrc=size=480x320:rate=30 -f lavfi -i sine -vf drawtext="text='%{localtime\:%X}':fontsize=20:fontcolor=white:x=7:y=7" -pix_fmt yuv420p -vcodec libx264 -vb 2000k -preset ultrafast -acodec aac -g 30 -f flv rtmp://nyc-canary-catalyst-0.livepeer.fun/live/e54e-fhvp-cgum-rd7a

I curl'd for the playlist in LON:

curl https://lon-canary-catalyst-0.livepeer.fun/hls/video+e54elimr0jpntzm4/index.m3u8

The above command hung for quite awhile (i.e. > 15s).

Then, it failed with with:

#EXTM3U
#EXT-X-ERROR: Stream open failed
#EXT-X-ENDLIST

Then, upon trying again, a valid playlist was returned:

#EXTM3U
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-STREAM-INF:CODECS="avc1.42c015,mp4a.40.2",RESOLUTION=480x320,FRAME-RATE=30,BANDWIDTH=1251765,AVERAGE-BANDWIDTH=1059186
0_1/index.m3u8?mTrack=0&iMsn=72&sessId=804883114
#EXT-X-STREAM-INF:CODECS="avc1.4d4020,mp4a.40.2",RESOLUTION=1280x854,FRAME-RATE=30,BANDWIDTH=1004702,AVERAGE-BANDWIDTH=850133
2_1/index.m3u8?mTrack=0&iMsn=72&sessId=804883114
#EXT-X-STREAM-INF:CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=854x570,FRAME-RATE=30,BANDWIDTH=1020022,AVERAGE-BANDWIDTH=863095
3_1/index.m3u8?mTrack=0&iMsn=72&sessId=804883114
#EXT-X-STREAM-INF:CODECS="avc1.4d401e,mp4a.40.2",RESOLUTION=640x426,FRAME-RATE=30,BANDWIDTH=531534,AVERAGE-BANDWIDTH=449759
4_1/index.m3u8?mTrack=0&iMsn=72&sessId=804883114
#EXT-X-STREAM-INF:CODECS="avc1.4d4015,mp4a.40.2",RESOLUTION=426x284,FRAME-RATE=30,BANDWIDTH=274435,AVERAGE-BANDWIDTH=232214
5_1/index.m3u8?mTrack=0&iMsn=72&sessId=804883114

[Thulinma]

Hmm... That sounds like maybe the internal reconnect mechanism isn't working properly for some reason. That ~15s wait time followed by error would indicate it was unable to access the stream buffers, but the successful request right after shows that the buffers are definitely there and filled. 🤔

[iameli]

Doing another round of testing, currently got all successes but some longish start times:

lax     3.82 secs
mdw     3.21 secs
nyc     2.65 secs
sao     6.05 secs
lon   130.76 millis
fra     2.58 secs
prg     9.92 secs
sin     4.40 secs

[iameli]

Logs for the 9.9 second startup in PRG are visible here: https://eu-metrics-monitoring.livepeer.live/grafana/explore?orgId=1&left=%5B%221661781435209%22,%221661781564369%22,%22Loki%22,%7B%22refId%22:%22A%22,%22expr%22:%22%7Bregion%3D%5C%22prg%5C%22,%20namespace%3D%5C%22canary%5C%22,%20container%3D%5C%22catalyst%5C%22%7D%20!%3D%20%5C%22vmagent%5C%22%20!%3D%20%5C%22API%20access%5C%22%22%7D%5D

[iameli]

Seeing 15s startup times in distant regions again even with the adjusted DVR setting. I'm putting together a little repeated automated test to give us some harder numbers.