meeb / tubesync

Syncs YouTube channels and playlists to a locally hosted media server

Hi

My GitHub profile hosts any public project that is polished enough to be released and used by others. Feel free to drop me an email, create issues or make contributions. A special massive thank you to anyone who sponsors me.

Cheers!

//meeb

🔗 meeb.org    |    📧 [email protected]

---

tubesync's Issues

Changing source settings does not modify existing files

For example, if I check "Copy thumbnails" after I did the initial save and download it should probably download them either right away or on the next scan. Same with changing the name format or directory.

Split workers into indexer and downloader jobs

Split indexing and downloading tasks into two different workers. This should help issues such as #34 and #32 as well as improve concurrency. This will require using background task named queues so the "tasks being worked on for this source" box may have to be removed as this feature is currently (ab)using the named task queue field.

enhancement: plug-ins for users to add new sites similar to how youtube-dl works

I've been working sporadically on a similar python script similar to tubesync, though far more rudimentary. I planned it to basically be a companion to youtube-dl that would scrape the same sites (or the few I'm interested in) and allow you to download the videos, instead of manually collecting them and sending them to youtube-dl. tubesync is far better. I'm a mediocre coder at best anyway. I looked through the FAQ. I saw there was a possibility of including other sites for this. I looked through the code a bit and to see if it modularized similar to how youtube-dl allows you to add more sites like plug-ins. It's fairly easy to just copy an existing one and modify it for a new site. I didn't look in depth but is the youtube specific part of tubesync like this? If it were, other users and devs could easily add new sites with an example template. Although, that may open the flood gates to being overwhelmed by users asking for assistance in development. Just a thought. I'd certainly be willing to contribute a few.

Refactor sync models and rework metadata handling

The sync app models are pretty unwieldy. They need a good tidy up and refactoring. Metadata should be split off and probably have some caching applied to avoid calling json.loads(...) excessively.

Logon Screen

Not sure if I missed it, but TubeSync seems to be missing an authentication option. Any chance of getting a logon option for security?

Custom media paths

Sources need to have custom media paths as an option. Media need to have custom media paths as an option. This probably can't use youtube-dls output formatting flags but one could probably chain off them to pre-populate some variables. Examples:

• /targetdir/[channel]/[year]/[title]-[format].[ext]
• /targetdir/[channel]/[format]/[filename].[ext]
• /targetdir/[channel]/[audio/video]/[filename].[ext]

etc. Probably needs to be user-settable string formatting.

This needs to be highly checked to avoid easy to inject common errors. Will need tests. Mentioned in issue #2 as well.

Only scan full metadata for each video once

The default youtube-dl extract_info() helper method crawls every video on every playlist or channel every time it's called. With large channels this can create thousands of indexing info requests. A closer look into the internals of youtube-dl is needed to have a way of indexing video IDs from the playlist overview pages without requesting each video. Later another deferred task can index the metadata for each video once at a slower rate. This should solve HTTP/429 (too many requests) issues when adding large numbers of big channels and annoying YouTube with thousands of duplicate requests per day.

Add option at source level to copy thumbnails

Plex does not support embedded metadata images in MKV containers, however, can add an option to copy over the media thumbnail image with the same filename as the media file but with a JEPG extension. This would allow the local media asset scanner in Plex (and other media servers) to pick up the correct thumbnail for media.

Some channels are not scheduled

At the moment I am monitoring 9 channels, but under Tasks I only see 3 index media tasks scheduled, all the other channel folder stay empty.
I got this a few times and used Reset Tasks to get it start downloading but it takes a very long time to reindex all channels.

Changing the channel settings seems to work but only once, the channels are downloaded but not added as an index media task.

Also, when a download failed, should it be added to the queue forever or does it give up after a few attempts, because I got a lot of failed downloads that aren't retried, I can make a new issue with those errors if you like.

I'm using the :latest tag because otherwise 'Reset Tasks' causes the 429 rate limit.

Files aren't being named properly

Below is the list and how its setup. I'm getting the first file named properly but files after that are weird symbols and letters after that.

Pull Up - 2020-03-22 - Pull Up - The Best Of [1080p-avc1-mp4a],mkv
PENT0U~L.mkv
PAX7MC~I.mkv

Type | YouTube playlist
-- | --
Name | Pull Up
Media items | 0
Key | PLhKmHfWT5YmKh7oyYbW-tyDwS3_23Kw64
Directory | /downloads/Pull Up/
Media format | Pull Up - {yyyy_mm_dd} - {title_full} [{format}].{ext}
Example filename | Pull Up - 2020-12-19 - Some Media Title Name [1080p-avc1-mp4a-60fps-hdr].mkv
Index schedule | Every 6 hours
Created | 2020-12-19 10:57:56
Last crawl | Never
Source resolution | 1080p (Full HD)
Source video codec | AVC1 (H.264)
Source audio codec | MP4A
Prefer 60FPS? |  
Prefer HDR? |  
Output extension | mkv
Fallback | Get next best resolution but at least HD
Copy thumbnails? |  
Write NFO? |  
Delete old media | No, keep forever

Automatic format retrieval option

If it's possible, it would be nice to see an option which automatically picks the best video resolution and audio using youtube-dl's -f bestvideo+bestaudio param instead of selecting manually an option in the source configuration

Perhaps this was the intention with the fallback. But i'm still selecting the highest known video resolution and an audio codec

PS nice project, just stumbled on it. Hope to replace my cronjob mess soon!

Pass through download progress to UI

Youtube-dl returns it's progress in the console. Would be nice to have this in the web ui too.

... - 29.5% of 1.59GiB at 13.65MiB/s, 01:23 remaining

Add playlist index to media format options

Add in a {playlist_index} to media format options so files can be saved into subdirectories even if indexed from a large channel with multiple playlists.

/ in video title

/ in video title creates video in another folder with the name after the /

Guide on correct Plex setup?

Is there a recommended way to setup Plex for this?

Mainly I'm after the Plex library settings to use. For example which agent and scanner.
Do we add the base directory or just video?

Fix YouTube channel validation

The validator is over-zealous and blocks valid "non-vanity" channels with IDs only. Examples:

• https://www.youtube.com/channel/LoNgIdHeErE
• https://www.youtube.com/channel/LoNgIdHeErE/videos

These need to be parsed properly in the add-source validator.

Re-host ffmpeg static build

Build process gets ffmpeg from johnvansickle.com, probably shouldn't spam it. Re-host ffmpeg somewhere.

Add source level options for subtitles

As raised in #1 there should be some options to obtain subtitles automatically.

Request: Make footer stick to the bottom of the window on pages that don't fit the window height

I basically have noticed this problem:

Footer not sticking:

Footer sticking:

According to this stack overflow post/answer, this isn't hard to accomplish, just a div tag and some CSS.

My implementation in the above picture actually requires 2 divs, one wrapping all of the content and one wrapping everything but the footer, otherwise the nav and main tags get spaced out as well.

Example:

<body>
    <div class="flex-wrapper">
        <div>
            <header></header>
            <nav></nav>
            <main></main>
        </div>
        <footer></footer>
    </div>
</body>

I would do this myself and submit a PR, but I have no experience with using SASS.

Downloads Fail - Error: "local variable 'vformat' referenced before assignment"

Hello - having an issue with both v0.5 and 0.6. Indexing works fine but the downloads always fail with the error -

Error: "local variable 'vformat' referenced before assignment"

I have manually selected the video & audio formats as mp4 - not sure if this is causing an issue?

Let me know what logs you need or anything else to look at.

Error when downloading videos with no downvotes

Couldn't download this video (https://www.youtube.com/watch?v=O7AXkEzX2VM&ab_channel=LoLEsportsVODsandHighlights) with no downvotes (at time of writing)

Traceback:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/background_task/tasks.py", line 43, in bg_runner
func(*args, **kwargs)
File "/app/sync/tasks.py", line 335, in download_media
write_text_file(media.nfopath, media.nfoxml)
File "/app/sync/models.py", line 1081, in nfoxml
votes.text = str(self.votes)
File "/app/sync/models.py", line 981, in votes
return upvotes + downvotes
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

Handle YouTube channels without vanity URLs

YouTube seems to have two URL channel formats which both work in varying situations:

• https://www.youtube.com/channel/ChAnNeLiDhErE
• https://www.youtube.com/c/vanitychannelnamehere

Channel IDs do not work for /c/ short names and short names do not work on /channel/ prefixes.

Unable to archive channel with vanity url

Hello, firstly, thank you making this project, it has saved me a lot of time and effort!

I am facing an issue where I am unable to add the following channel as a source, I tried adding it with both options, channel id and url.
https://www.youtube.com/user/SethEverman/videos

The issue is when changing the url to the required format (https://www.youtube.com/c/SethEverman), it results in an error, the page doesn't exist when using the /c/ url. It appears to be a similar issue to #9

Bulk add feature

Request: I'd like to see an option to mass add channels, etc.

Reason: I would like to add 500+ channels but doing so one by one is going to take way too long.

X_FRAME_OPTIONS is defaulted to DENY

In order to embed WebSync in something like Organizr, X_FRAME_OPTIONS = 'SAMEORIGIN' should probably be used in settings.py to allow iframe embedding

Textbox Misaligned

Not sure how to explain this issue but I have to click higher up in the middle of the text box to the upper half of it even the upper outside in order to select and change the text options.

Clean file name of "dangerous" characters

I've been choosing a directory name, and then choosing this as my file name
Season {yyyy}/{source_full} - {yyyy_mm_dd} - {title_full} ({key}).{ext}

For the most part it works well, until the {title_full} includes characters such as | or / or " or ' or \

Can you offer a toggle or option to strip characters not safe for the filesystem from the generated filenames, excluding characters actually specified in the Media Format option? Cleaning all the variables before being used in the generated file name should do the trick.

Updating YoutubeDL Manually?

Might be a dumb question but is there a way to have tubesync use a more up to date version of youtube or put in fields like cookies.txt into it?

Request: local_settings.py example

Thanks for putting this together!

Is there an example of how local_settings.py can be used to override the default settings?

Specifically I'm interested in providing additional parameters to youtube-dl (eg. subtitles) via the YOUTUBE_DEFAULTS attribute.

500 Internal Server Error when adding new channel

I was able to successfully add 4 channels yesterday, but now I get a 500 error when attempting to add a new channel.

`172.18.0.1 - - [18/Dec/2020:08:19:36 -0500] "POST /source-add HTTP/1.1" 500 268 "http://192.168.7.238:4848/source-add?source_type=c&key=SousVideEverything&name=SousVideEverything&directory=SousVideEverything" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.101 Safari/537.36"

172.18.0.1 - - [18/Dec/2020:08:19:36 -0500] "GET /favicon.ico HTTP/1.1" 302 0 "http://192.168.7.238:4848/source-add" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.101 Safari/537.36"

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.

Failed to retrieve tasks. Database unreachable.`

Running v 0.3.

I looked in my config folder for a log file but didnt see one - is there a better way to grab logs than via the CLI?

NFO Support

To help media players to index these videos, it might be cool to allow TubeSync to write NFO files for each piece of media

The Kodi standard is probably best: https://kodi.wiki/view/NFO_files/TV_shows

Subfolder support

Is there planned support for subfolders?

ex: http:///tubesync/

Download tasks never start

I have a bunch of channels added, and it queues things up, but it never downloads anything.

When going to one of the episodes, it has all the pertinent details but the "Task will run..." time/date is in the past. It never downloaded the video:

Docker logs show that it says it's downloading files but no files hit the drive (snippet, the only other log entries were webserver access logs):

2020-12-18 11:17:33,588 [tubesync/DEBUG] [download] Downloading video 17 of 181
2020-12-18 11:17:33,589 [tubesync/DEBUG] [youtube] 6TpwIxIU7qM: Downloading webpage
2020-12-18 11:17:34,579 [tubesync/DEBUG] [download] Downloading video 18 of 181
2020-12-18 11:17:34,579 [tubesync/DEBUG] [youtube] fnQ01hViDNo: Downloading webpage
2020-12-18 11:17:35,565 [tubesync/DEBUG] [download] Downloading video 19 of 181
2020-12-18 11:17:35,565 [tubesync/DEBUG] [youtube] Zcj8ZG_V_14: Downloading webpage
2020-12-18 11:17:36,350 [tubesync/DEBUG] [download] Downloading video 20 of 181
2020-12-18 11:17:36,350 [tubesync/DEBUG] [youtube] u9TYS_J1udo: Downloading webpage
2020-12-18 11:17:37,575 [tubesync/DEBUG] [download] Downloading video 21 of 181
2020-12-18 11:17:37,575 [tubesync/DEBUG] [youtube] QfQf4BZRCmo: Downloading webpage

what would be the best way to go about troubleshooting this?

Look into switching to async background workers and increasing the number of workers

If one were to add a bunch of sources with many videos and a low index interval it's possible the single foreground worker will spend its entire time just indexing media. One option here is to switch to using async background workers and increase the number of workers. The downside here is complexity of logging to console as events will be reported out of order (so a per-source log file or similar debugging system may be required) as well as improvements required to the race condition handling of assigning tasks to workers. This can get messy and is beyond the scope of what the upstream background_task library currently provides out of the box in some cases. Still, this probably needs to be looked into further for people with huge numbers of media to index.

Request: Skip/Import existing files

I have done a lot of downloading with youtube-dl already, most with different filenames that what tubesync allows so I can't match the new filename, but all with the [videoid] in the name ( for plex integration ), would it be possible to match on the videoid in the filename and mark the file as downloaded (Maybe set the image/xml data to that filename if at all possible)?

Change defaults

I cant seem to find where defaults are kept. Id like to set defaults for certain things when I add a new channel or playlist like mp4 etc etc.

also, I keep my channels seperated in subdirs (woodworking, welding, gardening) if i add that as the folder to download to, then add {source}/ to filename, the next wood working channel i add throws an error that another channel is in woodworking. basically it looks like

ytvids/woodworking/channel1
ytvids/woodworking/channel2
ytvids/gardening/channel1

my first level dir is basically a category, theis helps when i add libraries in plex or jellyfin as it set to woodworking and has all my ww channels.

Any idea how to do similiar without having to add it in the filename field?

Thanks

Downloading stopped at some errors, doesn't restart anymore

Installed v0.8 today. Added two channels and it started indexing and downloading thumbnails. After some time, it started downloading the actual videos, all was good.

When I checked a few hours later, it had stopped downloading. It showed some errors for three videos. But there are still some 500+ tasks in the list with scheduling "running immediately".

I clicked on the three errors and either skipped the media or scheduled to try again. But still, the other tasks won't start again. I restarted the container, also without luck.

How can I get the workers to continue with the downloading?

Below is the last part of the log. The config and download directories are mounted directories from my NAS. I assume that's why there's a chown Operation not permitted message, but the docker should have full read/write access on those files.

chown: changing ownership of '/config/media/thumbs/18': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/f4/f488e34d-ee6d-4dc0-b60b-8d4d835d7c35.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/f4/f4f81a9f-da06-4c54-adcf-4082d55511fb.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/f4/f44ae76b-f2e2-4bd7-b1e7-0f2598afb22b.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/f4': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/1f/1f21397a-a121-4926-95bc-71277a9e7452.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/1f/1fe22f32-de98-4095-80da-215497b6959f.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/1f/1f2a3032-d066-453b-9910-cce94f20866f.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/1f/1f5c1b6e-c99a-402e-a8f1-ea984d0ab5a3.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/1f/1f0872cd-ac53-4551-927d-8d2a949a596b.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/1f': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/fd/fdf8bbc1-d5c8-4a1e-aaa6-61d609c66f68.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/fd/fd4c624b-eef0-4fd8-b02b-dd7abb06633f.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/fd/fd11fe87-417a-4dc1-9d10-e59ce1017feb.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/fd/fdfd69e2-783c-484b-8f02-a4afe93ab39a.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/fd': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/0e/0e55a1af-278d-4454-b7f1-20a289c9a270.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/0e/0e9c2ebe-71e7-450e-b249-9262220be968.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/0e': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/67/67a86c09-3d03-4340-a119-1aba6589cdde.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/67': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/9d/9dbecbf1-04ab-4b85-8fb1-c3ad205f8f3d.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/9d/9d23a1f8-be1a-4589-828c-259dc53caaf0.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/9d/9d8cdb51-eeac-4d41-acb3-54e3c3b5594d.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/9d': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/3b/3b1a1062-cfd8-4015-bf2c-96c2700bd54f.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/3b/3beefd97-436e-4b86-8006-a4802fdb14e9.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/3b/3b659a48-643e-4e7e-b9fa-bf677bea6b8c.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/3b/3b9550f4-1dc7-4987-a38a-a751cb709f4f.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/3b': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/d1/d194b2e9-f017-471a-8bbc-9b987cf0d083.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/d1': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/2f/2f8a0c36-2243-4a9f-ae73-3ec2831fefcd.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/2f/2f6f378b-4b56-4b10-a721-7d178c7c6b7d.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/2f/2f124dd9-bd83-4d6b-83cf-b093a2cf0f63.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/2f/2fb28d06-8915-44dc-a51f-0010ea152d57.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/2f': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/83/83edb590-c800-40d9-8631-e9ebfeb0779d.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/83/836c87a0-4f87-44e3-867d-117eff770555.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/83': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/08/08128616-64f1-4f2b-8d0b-8f2ddae0c2d7.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/08/082dcb34-6bd4-4428-8af6-a88b715e3a6f.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/08/08d97c91-352a-4614-a856-76b390dfeafe.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/08': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/a3/a3ba83bf-cd35-47ed-b9ca-077912f7c429.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/a3': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/cb/cb0c7b15-84fc-4773-9236-a9a448d77c7c.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/cb/cb581c38-5366-4b1c-a3c2-3f2748bcc091.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/cb/cbbd07a1-54d1-4d50-af8b-2d9b782225b7.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/cb/cb1e1879-966a-4d23-8887-5fc0dd970687.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/cb': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/38/3890df88-681f-446f-859c-90dfd0c0fc80.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/38': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/0a/0a9e7919-a3b1-4668-b626-cad8b8f02bb3.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/0a/0a0e8455-c9e6-41e5-ab86-e904433c40a9.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/0a/0a35d89b-bd09-47fa-8af2-87f2c47f667a.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/0a': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/20/20a8ae50-8708-4d32-8a9f-91056c47bcf8.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/20': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/5c/5c92d7ff-5f97-4dd9-af69-cdcb408e0a1a.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/5c/5c2981ca-dc43-4ea5-b71e-0b1b92d0f020.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/5c': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/89/89530a91-e31c-44a4-8562-8fe69a680dee.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/89/895bec01-52ac-45ac-9bd0-76ee1f763841.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/89': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/ad/ad112e5d-5252-4eb7-9129-a8e7f16d91ef.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/ad/adf1ea2c-a85e-46e0-a8ac-6434bc6f2ab2.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/ad/ad908a76-22f1-43f1-a56b-b4d848f0fdfe.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/ad/ad1275b5-aab9-466e-953e-b428e6bd449a.jpg': Operation not permitted,
chown: changing ownership of '/config/media/thumbs/ad': Operation not permitted,
chown: changing ownership of '/config/media/thumbs': Operation not permitted,
chown: changing ownership of '/config/media': Operation not permitted,
chown: changing ownership of '/config/cache/youtube-sigfuncs/js_6eebf7aa_102.json': Operation not permitted,
chown: changing ownership of '/config/cache/youtube-sigfuncs/js_6eebf7aa_106.json': Operation not permitted,
chown: changing ownership of '/config/cache/youtube-sigfuncs': Operation not permitted,
chown: changing ownership of '/config/cache': Operation not permitted,
chown: changing ownership of '/config': Operation not permitted,
Operations to perform:,
  Apply all migrations: admin, auth, background_task, contenttypes, sessions, sync,
Running migrations:,
  No migrations to apply.,
  Your models have changes that are not yet reflected in a migration, and so won't be applied.,
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.,
[cont-init.d] 10-tubesync: exited 0.,
[cont-init.d] done.,
[services.d] starting services,
[services.d] done.,
[2021-02-17 23:17:16 +0800] [223] [INFO] Starting gunicorn 20.0.4,
[2021-02-17 23:17:16 +0800] [223] [INFO] Listening at: http://127.0.0.1:8080 (223),
[2021-02-17 23:17:16 +0800] [223] [INFO] Using worker: sync,
[2021-02-17 23:17:16 +0800] [249] [INFO] Booting worker with pid: 249,
[2021-02-17 23:17:16 +0800] [250] [INFO] Booting worker with pid: 250,
[2021-02-17 23:17:16 +0800] [251] [INFO] Booting worker with pid: 251,
192.168.7.10 - - [17/Feb/2021:23:21:33 +0800] "GET / HTTP/1.1" 200 3640 "http://192.168.7.14:4848/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0",
192.168.7.10 - - [17/Feb/2021:23:21:33 +0800] "GET /static/styles/tubesync.css HTTP/1.1" 200 34477 "http://192.168.7.14:4848/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0",
192.168.7.10 - - [17/Feb/2021:23:21:33 +0800] "GET /static/images/favicon.ico HTTP/1.1" 499 0 "http://192.168.7.14:4848/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0",
192.168.7.10 - - [17/Feb/2021:23:22:18 +0800] "GET /tasks HTTP/1.1" 200 28144 "http://192.168.7.14:4848/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0",
192.168.7.10 - - [17/Feb/2021:23:32:33 +0800] "GET /tasks HTTP/1.1" 200 28144 "http://192.168.7.14:4848/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0",
192.168.7.10 - - [17/Feb/2021:23:32:33 +0800] "GET /static/styles/tubesync.css HTTP/1.1" 200 34477 "http://192.168.7.14:4848/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0",
192.168.7.10 - - [17/Feb/2021:23:32:33 +0800] "GET /static/images/favicon.ico HTTP/1.1" 499 0 "http://192.168.7.14:4848/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0",

How to pay you?

Hey there,

you really scratch an itch with your project!
Installation and Usage has worked without any problems.
Since you are providing value to my life, I want that I can rely on a stable development.
Do you want to enable Github Sponsorship or an alternative to pay you?

Don't grab all pages if download cap is enabled

On initial index if "Download cap" is set then pages should only be fetched until it hits that the cap instead of fetching every single page of videos.

is older than cap age 2021-01-18 07:55:30.639997+00:00, skipping

I'd like to avoid seeing this over and over in my logs if possible.

Bug: Error when editing existing media server

Have plex media server added.
Go to media server information page.
Click on "Edit Media Server" button at bottom of page.
Get the following error:

500 - Internal Server Error
Your request caused an internal server error. This has been logged and our developers will implement a fix shortly.

For reference: running TubeSync 0.6 in Docker on Ubuntu

Youtube Channel errors when being indexed

Hey! Thanks for this great app! I got things working with a few sources and everything was working fine but now it's coming up with the below error for each source.

Index media from source "Youtube Channel", attempted 2 times
Error: "Youtube Channel" (ID: 'ChannelID') returned no media to index, is the source key valid? Check the source configuration is correct and that the source is reachable"

Health Check Failing with Permission Denied

Running the Docker image (pulled as ghcr.io/meeb/tubesync:v0.8) the container's Health Check fails with:

/bin/sh: 1: /app/healthcheck.py: Permission denied

Upon further investigation, the /app/healthcheck.py script does not appear to have execute permission:

# ls -la /app/healthcheck.py
-rw-r--r-- 1 root root 705 Jan 20 01:14 /app/healthcheck.py
#

Setting the permission manually inside the running container (with chmod +x /app/healthcheck.py) seems to resolve the problem.

Separate {yyyy_mm_dd}

Can we get separation of the date? so instead of {yyyy_mm_dd} {yyyy} {mm} {dd}

jobs arent being added.

I added about 20 sources and set them to 24hrs refresh. Out of the sources only one source ever gets updated. i have no indexing jobs for any of the other sources. any ideas?

videos dir

To really work like a pvr and for bigger collections it really needs the ability to ouput channels to sperate directories or use youtube-dl filename mask. Great work for sure and im liking the interface for sure.

Request: adding large channel, limit by date

Might be a nice feature for when a user is adding a large channel. Give a date field that will be checked against when downloading. For example, if you add a channel with thousands of videos over a number of years, be able to only download videos posted since a specific date. This would also allow people to select the current date if they only want to get videos going forward and not any of the ones that have already been posted. I would think default should still be "all videos" though.

Rework source validation

Currently the add a source flow is tightly restricted and this should be relaxed to allow freeform URLs to be entered which are properly parsed to detect a valid source type. This is the first step in resolving additional YouTube URL formats (e.g. #30) and for future support of other sources that are added to youtube-dl.

The current "add a button to add a source type" UI format isn't flexible enough to add a button per source. There's also more YouTube URL formats than first planned for. Adding a freeform URL box would resolve this, at the expense of writing moderately more complicated URL parsers.

add command line arguments

How do you add extra arguments to youtube-dl?

I am getting rate limited by google because the indexing is trying to pull all the videos

I want to add youtube-dl --dateafter now-{Download cap} to limit the number of items being queried.

per yt-dl docs: https://github.com/ytdl-org/youtube-dl#video-selection-1

Request: add toggle to hide skipped/deleted media

A thought that might be good from a usability standpoint: add a toggle/option to be able to hide media items that have been deleted/marked as skipped.

I could see a situation where a youtube channel has some videos that are worth keeping, but a bunch of short ones in between that aren't. Hiding those that have been removed so that you can only see the ones that have been "kept" could be helpful. Setting it as an option that you can turn on or off as needed would probably work best in case someone wants to go back and re-enable certain videos that have been deleted and skipped.

TypeError: 'bool' object is not subscriptable

I'm testing this out and I am getting errors for 3 of the videos with the same error from one channel: TypeError: 'bool' object is not subscriptable for the acodec. Somehow it's not picking up the MP4A fallback format for these files.

The videos are as follows:

• Video 1
• Video 2
• Video 3

Edit to add: I'm running v0.7 via Docker.

Docker log output is as follows:

tubesync         | 2020-12-31T06:45:37.271334737Z Rescheduling Downloading media for "Realism Gets Even More Real: Crash Course Theater #32"
tubesync         | 2020-12-31T06:45:37.271384162Z Traceback (most recent call last):
tubesync         | 2020-12-31T06:45:37.271394256Z   File "/usr/local/lib/python3.7/dist-packages/background_task/tasks.py", line 43, in bg_runner
tubesync         | 2020-12-31T06:45:37.271404175Z     func(*args, **kwargs)
tubesync         | 2020-12-31T06:45:37.271411219Z   File "/app/sync/tasks.py", line 285, in download_media
tubesync         | 2020-12-31T06:45:37.271418468Z     filepath = media.filepath
tubesync         | 2020-12-31T06:45:37.271425576Z   File "/app/sync/models.py", line 1043, in filepath
tubesync         | 2020-12-31T06:45:37.271432561Z     return self.source.directory_path / self.filename
tubesync         | 2020-12-31T06:45:37.271439616Z   File "/app/sync/models.py", line 1010, in filename
tubesync         | 2020-12-31T06:45:37.271448879Z     media_details = self.format_dict
tubesync         | 2020-12-31T06:45:37.271458495Z   File "/app/sync/models.py", line 879, in format_dict
tubesync         | 2020-12-31T06:45:37.271465465Z     display_format = self.get_display_format(format_str)
tubesync         | 2020-12-31T06:45:37.271472406Z   File "/app/sync/models.py", line 841, in get_display_format
tubesync         | 2020-12-31T06:45:37.271481093Z     acodec = aformat['acodec'].lower()
tubesync         | 2020-12-31T06:45:37.271488194Z TypeError: 'bool' object is not subscriptable
tubesync         | 2020-12-31T06:45:37.279256328Z Rescheduling task Downloading media for "Realism Gets Even More Real: Crash Course Theater #32" for 0:10:30 later at 2020-12-31 06:56:07.278999+00:00
tubesync         | 2020-12-31T06:45:37.322813258Z Rescheduling Downloading media for "Hot Dog Contest 2014"
tubesync         | 2020-12-31T06:45:37.322875764Z Traceback (most recent call last):
tubesync         | 2020-12-31T06:45:37.322885741Z   File "/usr/local/lib/python3.7/dist-packages/background_task/tasks.py", line 43, in bg_runner
tubesync         | 2020-12-31T06:45:37.322894881Z     func(*args, **kwargs)
tubesync         | 2020-12-31T06:45:37.322901957Z   File "/app/sync/tasks.py", line 285, in download_media
tubesync         | 2020-12-31T06:45:37.322909094Z     filepath = media.filepath
tubesync         | 2020-12-31T06:45:37.322919566Z   File "/app/sync/models.py", line 1043, in filepath
tubesync         | 2020-12-31T06:45:37.322926625Z     return self.source.directory_path / self.filename
tubesync         | 2020-12-31T06:45:37.322933594Z   File "/app/sync/models.py", line 1010, in filename
tubesync         | 2020-12-31T06:45:37.322940517Z     media_details = self.format_dict
tubesync         | 2020-12-31T06:45:37.322947345Z   File "/app/sync/models.py", line 879, in format_dict
tubesync         | 2020-12-31T06:45:37.322955108Z     display_format = self.get_display_format(format_str)
tubesync         | 2020-12-31T06:45:37.322962066Z   File "/app/sync/models.py", line 841, in get_display_format
tubesync         | 2020-12-31T06:45:37.322969351Z     acodec = aformat['acodec'].lower()
tubesync         | 2020-12-31T06:45:37.322976192Z TypeError: 'bool' object is not subscriptable
tubesync         | 2020-12-31T06:45:37.332229543Z Rescheduling task Downloading media for "Hot Dog Contest 2014" for 0:10:30 later at 2020-12-31 06:56:07.332021+00:00
tubesync         | 2020-12-31T06:45:37.362685274Z Rescheduling Downloading media for "Crash Course Surveys"
tubesync         | 2020-12-31T06:45:37.362736361Z Traceback (most recent call last):
tubesync         | 2020-12-31T06:45:37.362745709Z   File "/usr/local/lib/python3.7/dist-packages/background_task/tasks.py", line 43, in bg_runner
tubesync         | 2020-12-31T06:45:37.362756157Z     func(*args, **kwargs)
tubesync         | 2020-12-31T06:45:37.362763633Z   File "/app/sync/tasks.py", line 285, in download_media
tubesync         | 2020-12-31T06:45:37.362771441Z     filepath = media.filepath
tubesync         | 2020-12-31T06:45:37.362778364Z   File "/app/sync/models.py", line 1043, in filepath
tubesync         | 2020-12-31T06:45:37.362786675Z     return self.source.directory_path / self.filename
tubesync         | 2020-12-31T06:45:37.362793510Z   File "/app/sync/models.py", line 1010, in filename
tubesync         | 2020-12-31T06:45:37.362800527Z     media_details = self.format_dict
tubesync         | 2020-12-31T06:45:37.362807322Z   File "/app/sync/models.py", line 879, in format_dict
tubesync         | 2020-12-31T06:45:37.362815939Z     display_format = self.get_display_format(format_str)
tubesync         | 2020-12-31T06:45:37.362822907Z   File "/app/sync/models.py", line 841, in get_display_format
tubesync         | 2020-12-31T06:45:37.362829974Z     acodec = aformat['acodec'].lower()
tubesync         | 2020-12-31T06:45:37.362836782Z TypeError: 'bool' object is not subscriptable
tubesync         | 2020-12-31T06:45:37.371240321Z Rescheduling task Downloading media for "Crash Course Surveys" for 0:10:30 later at 2020-12-31 06:56:07.371013+00:00

My settings for this channel are as follows: