After Upgrade, Older Videos Stopped Playing; Files Missing From Object Storage about peertube

[ChanoSan]

Update: I attempted to force HLS transcoding via the Administration interface, with success. However, the subsequent move to object storage failed.

Peertube job list showed it as DELAYED initially, but now shows it as FAILED. The logs for the job shows the following info:

Job: 84 Type: move-to-object-storage Processed on 7/10/24, 2:14:42.681 PM Finished on 7/10/24, 2:14:43.002 PM{ "videoUUID": "c14ae3f5-b44c-4547-b3b7-7a4be1fca095", "isNewVideo": false, "previousVideoState": 2 }TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received null at new NodeError (node:internal/errors:405:5) at validateString (node:internal/validators:162:11) at join (node:path:1175:7) at updateTorrentMetadata (file:///var/www/peertube/versions/peertube-v6.1.0/dist/core/helpers/webtorrent.js:101:28) at onFileMoved (file:///var/www/peertube/versions/peertube-v6.1.0/dist/core/lib/job-queue/handlers/move-to-object-storage.js:62:11) at moveWebVideoFiles (file:///var/www/peertube/versions/peertube-v6.1.0/dist/core/lib/job-queue/handlers/move-to-object-storage.js:41:15) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async moveToJob (file:///var/www/peertube/versions/peertube-v6.1.0/dist/core/lib/job-queue/handlers/shared/move-video.js:19:13) at async Object.processMoveToObjectStorage [as move-to-object-storage] (file:///var/www/peertube/versions/peertube-v6.1.0/dist/core/lib/job-queue/handlers/move-to-object-storage.js:16:5) at async Object.wrapPromiseFun (file:///var/www/peertube/versions/peertube-v6.1.0/dist/core/lib/plugins/hooks.js:8:24) at async Worker.processJob (/var/www/peertube/versions/peertube-v6.1.0/node_modules/bullmq/dist/cjs/classes/worker.js:395:28) at async Worker.retryIfFailed (/var/www/peertube/versions/peertube-v6.1.0/node_modules/bullmq/dist/cjs/classes/worker.js:584:24)

[Chocobozzz]

Hi,

Can you paste a video URL example?

[ChanoSan]

The following is a video that I forced HLS Transcode, but which failed moving to object storage:

https://peertube.anon-kenkai.com/w/pSnVKiGdLHcr6F2TycZgvg

This is another video stuck on Awaiting Transcoding:

https://peertube.anon-kenkai.com/w/ornoFWrVpFiZXcocyjHkrT

Here is a video that I haven't done anything to, it just won't play (and appears not to exist in object storage):

https://peertube.anon-kenkai.com/w/sJqeUW79rkYFx5ubLbatPm

...for reference, this is a video uploaded after upgrading to 6.1.0 which plays normally and exists in object storage:

https://peertube.anon-kenkai.com/w/4FpqkEbGtWJSYWdX1oYMYy

...and lastly here's an old video stored locally on the PeerTube server which also plays normally:

https://peertube.anon-kenkai.com/w/sjN7JJmuT6c7N7Cs2RYZGK

[ChanoSan]

As an addendum, I tried using CLI commands from /var/www/peertube/peertube-latest as described in the docs here, but it only recreated the job which failed again with the same error.

Right now, I'm in the odd position where two videos on my server are perfectly playable, but they don't count as "Published", and as such don't appear in the Local Videos or Channels view. They seem to be stuck in limbo, either "Awaiting Transcoding" that never comes, or else being stuck at "Move to Object Storage Failed", and thus not counting as published either.

I do wonder if disabling object storage in production.yaml and then replacing the video files might force the transcoded videos to be stored locally and published, at which point I could re-enable object storage and use the CLI to move them back over. I hope it doesn't come to that, but I can do it if I have to.

[ChanoSan]

Further followup: After seeing this comment from @Chocobozzz in #4659 , I was able to go into the database and revert videos stuck in "Move to External Storage Failed" to local storage and the video now displays in Local Videos without issue. Later today, I'll be attempting to move it to object storage via the CLI and see if it produces the same error as before.

[Chocobozzz]

From what I see in your videos metadata the web video is missing a torrent file (this situation should not happen, it's the reason of the error).

This case should now be correctly handled by c5fa9fe

To fix your issue, try to run HLS transcoding and web video transcoding on all your videos. If you don't need/want the web video version, then just run HLS transcoding and remove the web video version from the web interface

[ChanoSan]

From what I see in your videos metadata the web video is missing a torrent file (this situation should not happen, it's the reason of the error).

This case should now be correctly handled by c5fa9fe

To fix your issue, try to run HLS transcoding and web video transcoding on all your videos. If you don't need/want the web video version, then just run HLS transcoding and remove the web video version from the web interface

Thankfully, this seems to work, though it seems very finicky about the order of operations. If I do HLS transcoding without doing web video first, the move to object storage encounters errors again.

For anybody experiencing my bizarre edge case in future, the best course of action seems to be:

1. Enable video re-uploads in settings
2. Edit the video, and use "Replace video file" to reupload a backup of the original. Ignore the error it throws up when the download completes
3. Go to Administration > Overview > Videos and find the video in question
4. Choose "Run web video transcoding" and let it finish.
5. Then "Run HLS Transcoding"
6. Videos should upload to object storage as expected without error
7. Optionally, delete Web videos when it's all done.

Thank you @Chocobozzz for the assist. The mystery of why the files vanished from object storage in the first place remains unsolved and will haunt me to the day I die, but at least I can fix things now.

[ChanoSan]

Closing as solved.