mxpv / podsync Goto Github PK
View Code? Open in Web Editor NEWTurn YouTube or Vimeo channels, users, or playlists into podcast feeds
License: MIT License
Turn YouTube or Vimeo channels, users, or playlists into podcast feeds
License: MIT License
Captcha challenges got you down? Not for long! Not any more!
These tools solve, with automation and no user interaction of any kind, 95% of captcha challenges. See issue 286 on https://github.com/google/recaptcha/ — wanting to avoid actual issue metadata linking, but it's the most hilariously appropriate issue ever created. ;P
(Captcha is bad, and Google should feel bad.)
hey max
thanks for the service. need a little help.
i signed up for your patreon so i could download youtube playlists into audio podcasts (which allow me to open other apps while the podcast is playing)
i'm trying to take this playlist and turn it into an audio podcast feed, but no matter what i try to do, it ends up being a video podcast feed. so any time i close the podcast app, the podcast stops playing
https://www.youtube.com/playlist?list=PLlATC9A2NkJ-he8_O7fwpwCjumG5je6Xu
any help much appreciated. thanks!
All workarounds that I did last few months are related to finding ways to bypass various limitations (API tokens has limited quotas, the number of download requests is limited). I believe that the current path forward is to switch to self hosted version, which would allow to not only use it normally, but add some neat features (like mp3 support).
This issue is to track progress:
Patreon post: https://www.patreon.com/posts/self-hosted-aka-31073377
https://podsync.net/ele0
failed to query videos (error 200 200 OK): json: cannot unmarshal bool into Go struct field EmbedSettings.color of type string
Hi, first, love your service, keep it going! But on iOS (iPhoneX) is almost imposible to copy generated link. Its just too long. Can you add copy to clipboard? Its posible, here is how: https://stackoverflow.com/questions/34045777/copy-to-clipboard-using-javascript-in-ios
Even though I select "high" quality on the podsync website before inputting my url, a 3 minute 1080p video on youtube will only download to a 15mb file size and looks pretty bad on my 6 inch phone screen.
I a feed where one of the videos has a copyright violation error, the updater skips all future files.
I believe this stems from https://github.com/mxpv/podsync/blob/master/cmd/podsync/updater.go#L95 which goes like:
if os.IsNotExist(err) {
// There is no file on disk, download episode
logger.Infof("! downloading episode %s", episode.VideoURL)
if output, err := u.downloader.Download(ctx, feedConfig, episode.VideoURL, episodePath); err != nil {
logger.WithError(err).Errorf("youtube-dl error: %s", output)
// YouTube might block host with HTTP Error 429: Too Many Requests
// We still need to generate XML, so just stop sending download requests and
// retry next time
break
}
downloaded++
} else {
// Episode already downloaded
logger.Debug("skipping download of episode")
}
I can see in the comments that the break
is there for a reason; I'm guessing that it's to cover HTTP error 429. However, the youtube-dl error is:
[youtube] xxxxxxxxxxx: Downloading webpage\n[youtube] xxxxxxxxxxx: Downloading video info webpage\nERROR: This video contains content from SME, who has blocked it on copyright grounds.\nSorry about that.\n"
I think that the errors need to be checked and while the 429 errors should stop the update process, other errors should be allowed to continue to the next file.
I have podsync running on my own server behind nginx, with https enabled.
This means it is available at "https://podsync.myserver.net". However, the xml feed contains references to "http://podsync.myserver.net:9090/", which does not work for two reasons: it does not use ssl, which is required on my server + port 9090 is not exposed, the "podsync" subdomain just redirects to it.
Running the cli version 2.0.1
I have the page size set to 5 but it went through downloading until it had done 50
I notice it lists the format and quality when it runs but it doesnt show the page size.
Is this abuy or am i misunderstanding?
INFO[2019-11-14T19:38:32Z] -> updating https://www.youtube.com/channel/UCabq3No3wXbs6Ut-Pux6SzA/videos feed_id=TryChannel format=video quality=high
INFO[2019-11-14T19:38:32Z] running listener at :8421
INFO[2019-11-14T19:38:33Z] ! downloading episode https://youtube.com/watch?v=cNi8Aaf568E episode_id=cNi8Aaf568E index=42
INFO[2019-11-14T19:38:51Z] ! downloading episode https://youtube.com/watch?v=E7bPGdWJrIs episode_id=E7bPGdWJrIs index=43
INFO[2019-11-14T19:39:10Z] ! downloading episode https://youtube.com/watch?v=C0uCAEfLA1w episode_id=C0uCAEfLA1w index=44
INFO[2019-11-14T19:39:53Z] ! downloading episode https://youtube.com/watch?v=_PQnkjgspNk episode_id=_PQnkjgspNk index=45
INFO[2019-11-14T19:40:32Z] ! downloading episode https://youtube.com/watch?v=lkG2u1Zua-Q episode_id=lkG2u1Zua-Q index=46
INFO[2019-11-14T19:40:55Z] ! downloading episode https://youtube.com/watch?v=qHz8hpxszdw episode_id=qHz8hpxszdw index=47
INFO[2019-11-14T19:41:10Z] ! downloading episode https://youtube.com/watch?v=iWAV7L-stxQ episode_id=iWAV7L-stxQ index=48
INFO[2019-11-14T19:41:28Z] ! downloading episode https://youtube.com/watch?v=YSjAbXHNn0I episode_id=YSjAbXHNn0I index=49
INFO[2019-11-14T19:41:48Z] successfully updated feed in 3m16.6572769s, downloaded: 8 episode(s), next update at 7:41AM
In castamatic iOS app suor feed won't be recognised, i suppose because it doesn't had the .xml at end, please is it possibile to have the same feed but with the .xml at the end?
Totally docker newbie here, so I maybe asking something obvious.
When I try to pull images, I get this error message:
ERROR: for resolver pull access denied for mxpv/podsync_lambda, repository does not exist or may require 'docker login'
If I am expected to generate my own image and push it to my own registry, can you share a quick recipe for accomplishing this?
thanks
TODO:
Not manually. Feeds are automatically deleted if not used for more than 90 days.
Podsync can't build a feed from just one video. You have to provide a link either to a channel, user, or playlist.
This is known issue. Unfortunately as of now I don't have any fix for this.
That happens because you're in a different region than Podsync. More details on this here. I've been working on multi-region support, fix is coming soon.
Feeds are updated every 10 minutes.
No, that would require "login with your YouTube account" on Podsync. Instead you might want to create custom playlist and add there videos you want to watch.
** Hi Max, thanks for this good service! One of my concern is: The podcasting file is too big. Take https://www.youtube.com/watch?v=XxXxxXXxX for example, each episode is about 45 minutes. I set the podcast to audio format and worst quality, it's still 900M each file: http://podsync.net/Xxxxx (actually the size doesn't change whatever format/quality I set)
Thanks Max! I opened the feed using "Pocket Casts" App on android, and it shows ~900M for each file. Now I using another app "Podcast Go", the size is normal. Another question, why the "length" in "enclosure" tag of the xml is very big, such as "500150000"?
None of my feeds are updating at this point. If I recreate them it will pull the more recent episodes, but then those recreated ones won't update either.
Postgres
from GCP
to RDS/Lightsail
DigitalOcean
to Lightsail
Would you consider using the maxresdefault.jpg thumbnails, instead of default.jpg thumbnails? So glad to have found your service!
Channel: https://www.youtube.com/channel/UCjM-3dwp2cToylYQsKzVYZA
Feed: http://podsync.net/JrWL
Response: Invalid page count
I would like to avoid scraping of my server by (optionally) eliminating the indexing of files from the root directory.
Current behaviour: GET / returns a page containing an index comprised of podcast folders and feeds. GET /podcastID returns a page containing a list of MP3s
Requested behavioue: GET / returns 404; GET /podcastID returns 404
Integrate let's encrypt support for embedded https.
golang.org/x/crypto/acme/autocert
looks like a good start for this. https://stackoverflow.com/questions/37321760/how-to-set-up-lets-encrypt-for-a-go-server-application
ref: #47
Runing podsync as docker.
When I run it, I see it downloads all the mp4 files.....so i naturually thought, hmm maybe I can put these in plex as a library. The easiest way I can think of doing this, is for each media file to add an nfo file.
i don't know go to save myself, but if I hacked around inside the container and added a method like buildpodcast (in updater.go) for nfo, and wrote out a new file each time, would it run (ie like python does), or is go compiled like c, So I'd have to rebuild the image?
I have set one of the channels' page_size to 10, and there were more than 10 new videos on the channel after the 1st query. After the 2nd update, 10 more new episodes were downloaded. The feed display the most recent 10 new episodes, but the previously downloaded files still remains in the downloaded folder. Is this expected behavior? I feel like if the xml file doesn't include the old downloaded files, they should be removed after the feed is generated during each query.
Recreation Steps:
From root directory, run sudo make build
Expected result:
Build logic from Makefile
Result:
make: *** No rule to make target 'build'. Stop.
It looks like there is no build command in Makefile currently.
SUBDIRS := cmd/api cmd/nginx
BINPATH := $(abspath ./bin)
GOLANGCI := $(BINPATH)/golangci-lint
.PHONY: push
push:
for d in
$(GOLANGCI):
curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b $(BINPATH) v1.17.1
$(GOLANGCI) --version
.PHONY: lint
lint: $(GOLANGCI)
$(GOLANGCI) run
.PHONY: test
test:
go test -short ./...
.PHONY: up
up:
docker-compose pull
docker-compose up -d
For example, http://podsync.net/DAgA (https://www.youtube.com/channel/UCdHZela1qTNM9iIw2YxenIw) which includes http://podsync.net/download/DAgA/K-kqyeVM_Qc.m4a (https://youtube.com/watch?v=K-kqyeVM_Qc). XML lists the duration as 00:14:39 and the fileSize as 307650000. The file downloads (either directly in Chrome or via https://play.pocketcasts.com) as an mp4 file of 13,960,760 bytes.
Mac’s Quick Look does not recognize the file as playable. Both QuickTime and and VLC (for VLC, Mac or Win) will open and play the file but dragging the playhead into the silence causes playback problems. iOS (e.g. Dropbox or Pocket Casts) will play the file but it’s half silence. In Chrome (Mac) or Edge (Win), the downloaded file has the correct duration and plays OK. Attempting to copy the file over a network on Windows with TeraCopy errors with the Status ’SetEndOfFile fails, try enabling “Use system write cache” option’.
I hope this helps.
2017-10-26 18:08:12 - (sanic)[ERROR]: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 503, in handle_request
response = await response
File "/app/ytdl.py", line 43, in download
json = r.json()
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 892, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2017-10-26 18:08:12 - (network)[INFO][172.18.0.5:53912]: GET http://ytdl:5002/download/youtube/HkvE_0k_2q0.mp4 500 144
2017-10-26 18:08:12 - (sanic)[ERROR]: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 503, in handle_request
response = await response
File "/app/ytdl.py", line 43, in download
json = r.json()
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 892, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
First of all, really like your work.
Are you planning on adding ARM support, so we can run Podsync on the Raspberry Pi and similar devices?
Costs
RDS: ~$30
ALB: ~ $40
EC2: ~$2x30
Other: ~$20
--
Total: $150
Deleted
Hi everyone,
there is an issue with sound and video output after subscribing to it using Podcast. Cover and names and lenght of the included items are loaded but after pressing play there is no audio and video output.
Podsync isn't updating new uploads from YouTube, not sure about other sites. It did sync once, but that was like after 36 hours. But after that one time, it haven't been updating. It's been 3 days (Aug 17) on my system since new content has been available through the feed.
Howdy! Just patron'd, love the service, wanted to investigate the quality settings (not well documented as to exactly what the difference is between low and high quality…) and can't log in with patreon… the podsync feed URL is a GitHub Gist containing the JSON message I get in Safari. Looks like a "www." vs. no "www." issue, as scrubbing "www." from the origin URL seems to have corrected the issue. Have a great day, and keep on' being awesome!
Does it clean up old episodes to save space? Documentation says nothing regarding cleanup. And if so, is it configurable? If not, it may be worth to add this feature. And make it configurable. best would be to be able to configure max files and max size per feed. If configured both, the first one hit would be effective. Global would also be nice.
Just found your awesome site! having a small issue though. Is there a way to reverse the playlist order? the playlist creator sorts it in ascending order.
Message (required)
Hi,
Thank you for such a wonderful service. I was just wondering if it were possible to sort out the Author attribution. Currently on my feedburner it says that it's 'Unknown' on Apple Podcasts, and we'd love it to say British Rationals which is the name of the channel.
Video title is being returned as "_" (an underscore).
...
<item>
<guid>https://dl.podsync.net/download/pzyl/T-lBMrjZ3_0.mp4</guid>
<title>_</title>
<link>https://www.youtube.com/watch?v=T-lBMrjZ3_0</link>
...
Example:
https://podsync.net/pzyl
Hello Maksym,
ok. Bummer! I’ve tried another service, which seems to be working with the live streamed clip, and access the file on google server properly http://www.listentoyoutube.com/ - maybe some way of checking how they access the file?
Also, its not working on iTunes. I tried it on another computer without any luck. Maybe some caching issues, i don’t know - but it won’t even start the ones the previously worked. Also tried another youtube-channel (https://www.youtube.com/user/ADDatWorkProductions + http://podsync.net/pKBV ) , and audio-version - but it won’t start. Here the log from iTunes
förval 10:52:22.696577 +0100 iTunes TCP Conn 0x600000380f70 started
förval 10:52:22.719424 +0100 iTunes TCP Conn 0x600000380f70 event 1. err: 0
förval 10:52:22.719453 +0100 iTunes TCP Conn 0x600000380f70 complete. fd: 42, err: 0
förval 10:52:22.719635 +0100 iTunes TCP Conn 0x600000380f70 starting SSL negotiation
förval 10:52:22.782344 +0100 iTunes TCP Conn 0x600000380f70 SSL Handshake DONE
förval 10:52:22.906139 +0100 mediaremoted Setting 23 supported commands for com.apple.iTunes
förval 10:52:22.906306 +0100 iTunes Suppressing kMRMediaRemoteSupportedCommandsDidChangeNotification -- client process did not ask to receive these notifications.
förval 10:52:22.953693 +0100 iTunes TCP Conn 0x600000380f70 canceled
förval 10:52:22.995070 +0100 iTunes Received Media Remote client notification kMRMediaRemoteNowPlayingApplicationPlaybackStateDidChangeNotification
förval 10:52:23.074284 +0100 iTunes ### PlayQueue::StartPlayingNextInternal: played off the end of the queue
förval 10:52:23.186577 +0100 mediaremoted Setting 23 supported commands for com.apple.iTunes
förval 10:52:23.186749 +0100 iTunes Suppressing kMRMediaRemoteSupportedCommandsDidChangeNotification -- client process did not ask to receive these notifications.
förval 10:52:23.191014 +0100 iTunes Received Media Remote client notification kMRMediaRemoteNowPlayingApplicationPlaybackStateDidChangeNotification
förval 10:52:23.197580 +0100 iTunes Setting identical now playing info, skipping update.
förval 10:52:23.197618 +0100 iTunes Clearing now playing info
förval 10:52:23.197642 +0100 iTunes Setting identical now playing info, skipping update.
förval 10:52:23.197661 +0100 iTunes Clearing now playing info
förval 10:52:23.197847 +0100 iTunes Clearing now playing info
förval 10:52:23.197878 +0100 iTunes Setting identical now playing info, skipping update.
förval 10:52:23.197901 +0100 iTunes Clearing now playing info
förval 10:52:23.197924 +0100 iTunes Setting identical now playing info, skipping update.
förval 10:52:23.197944 +0100 iTunes Clearing now playing info
I use the docker version to generate a feed from youtube channel in audio format (quality = "high" format = "audio"), the files are stored in a folder (data_dir="/app/data/") and have the right name and extension (.mp3), but the files aren't mp3 anche can't be played.
I made some test in macOS and the error seams in youtube-dl
this command produce wrong file:
youtube-dl --extract-audio --audio-format mp3 --format bestaudio --output data/abc/TA-c16e_zZ8.mp3 https://youtube.com/watch\?v\=TA-c16e_zZ8
this command without output param produce the right mp3 but in wrong location:
youtube-dl --extract-audio --audio-format mp3 --format bestaudio https://youtube.com/watch\?v\=TA-c16e_zZ8
@pod_sync and here is how it looks (bug)
https://pbs.twimg.com/media/C1Y90alXgAAEcpK.jpg
is it just me or has anyone else been having issues with youtube channels converted to a podcast feed? new items are still being added but they don't play majority of the time. sometimes if i keep hitting play over and over it'll work eventually.
edit: nevermind, saw the tweet, seems this is a known issue.
The Podsync repo is missing a LICENSE file in the root of the repo.
I've just started using your service to download audio podcasts, but playback only works when streaming, and not when i've downloaded the episode.
I use Pocket Casts (version 6.5) on an iPhone 6 (iOs 10.2.1), and the youtube channel is The Ling Space (https://www.youtube.com/user/thelingspace http://podsync.net/LwXl).Here's exactly what happens when I try downloading episodes:
The episode seems to download just fine.
When I click the play button, the episode opens but nothing happens. Then I get this message if I try to play the episode again: "Playback failed. The episode might be corrupted, but you can try to play it again."
The duration of the episodes often seems to be wrong. For example it says this episode [ https://www.youtube.com/watch?v=8Xi81H80J-w ] is 20 min long, but it's actually just 10 min long. This mistake is corrected when I hit the play button.
I am running the CLI on port 8421 but the xml feed generated doesn't include this in the path to the video.
This generate a not found error when i try to play it
So
http://mydomain.homeip.net/TryChannel/IWihhlAW-2s.mp4
need to be
There are episodes that come out, for instance, every Friday at 8 pm.
So there is no reason to check for updates every 6 hours and drain API tokens.
Instead we can specify cron expression to check for update every Friday at 8:15pm
I have set up my own self-hosted version to track a youtube channel.
I've set it up as "low quality audio". I see that the episodes are downloaded in mp3 format.
However, the resulting xml-file for the feed contains references to m4a files, hence the feed does not work in my podcast app, because it fails with "file not found". The filenames are correct, just the extsenion is wrong.
Podsync version: commit=7c1b1046f19fee5fa85bb34c0f7edec7013d55f7 date="2019-11-14T06:56:06Z" version=2.0.1
using youtube-dl 2019.01.17
using ffmpeg ffmpeg version 4.1.4-1~deb10u1
I am having an issue where some episodes in a feed won't ever download. This has been happening to only certain episodes for about two weeks now. I've tried generating new versions of the feeds and different ways of downloading them and the problem persists, always with the same videos while others work fine.
What happens is they sit and try to download for an extended period of time, never make progress, and then eventually error out.
The Tested Youtube channel has multiple examples: https://podsync.net/1q6W2ZNf5
Specifically, the (currently most recent) episode titled "Cutting it Close - Still Untitled: The Adam Savage Project" is one of the ones that won't download.
We need to return Vimeo back
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.