endofhome / patreon-downloader

Automatic downloading, tagging and organising of audio files posted by Patreon creators

Patreon downloader

Summary

A collection of scripts to enable automatic downloading, tagging and organising of audio files posted by Patreon creators. The script uses Puppeteer to control an instance of the Chromium browser. A Patreon account is required, and if you want to access subscription-only content, you will need to pay for it. This script purely replaces a manual process with an automated one. It can be run on a cron job or by some other scheduling mechanism.

The script supports downloading of mp3 and wav files. mp3 files will also have a comment (COMM) id3v2 tag added, the content of which will be the text content of the blog post, followed by any tags set by the creator. The post's associated image will also be embeded as album art in the file. As wav does not officially support such metadata, no such tagging/embedding will occur. Instead a cue sheet will be written and transferred along with the audio and artwork files.

The script keeps track of any files downloaded in a text file and will not re-download them. Once files have been downloaded they will be moved to a final destination directory (for me, a hard disk attached to my media centre), either on the same machine using mv or elsewhere using scp.

Once the file is downloaded, an update of the audio library in a Kodi installation is triggered (given the correct configuration).

Dependencies

• nodeJS
• cURL
• id3 A.K.A. id3mtag (do not be confused by id3 on Ubuntu systems, this is not the same tool)
• eyeD3 (requires Python 2.7+ and libmagic)

To install id3mtag on macOS, refer to my blog post at https://endofhome.github.io/2019/01/11/compiling_id3mtag_for_macos.html

eyeD3 can be installed via pip: pip install eyeD3 It requires a working installation of libmagic. On macOS this can easily be installed via Homebrew: brew install libmagic, or on Ubuntu, apt-get install libmagic.

Configuration

As well as the required environment variables (listed below), the script requires a logged-in session cookie. I dumped my cookies to a JSON file using a browser extension. The only cookie required (at the time of writing) is session_id for the domain .patreon.com. The script requires the session cookie to be stored as a JSON object inside a JSON array in a file named cookies.json at root level.

Environment variables:

Variable	Description
PATREON_ARTIST	The Patreon artist/creator that you are supporting, as per the URL of their Patreon page
PATREON_ARTIST_NAME	The name of the artist/creator
DESTINATION_DIRECTORY	The directory you want to save your files in. For me, a directory in a disk attached to my media centre. This directory doesn't have to be on the same machine as you are running the script on, if you have an ssh key stored the script will be able to use scp to transfer the files
DESTINATION_MACHINE_NAME	Name of the machine where the destination directory exists
DESTINATION_MACHINE_USERNAME	Username for the account you will use to scp into the destination machine
DESTINATION_MACHINE_HOST	Host for the destination machine (used for scp)
KODI_USERNAME	Username for Kodi installation
KODI_PASSWORD	Password for Kodi installation
KODI_PORT	Port for Kodi installation

How do I run it?

npm i to install the nodeJS dependencies.

Install other dependencies mentioned above, and ensure the cookies.json file and necessary environment variables are present.

./run.sh will start the process.

patreon-downloader's Issues

Error: Protocol error (Network.deleteCookies): Invalid parameters name: string value expected

We get the error:

Session has been loaded in the browser
Starting Chromium and looking for new songs to download...
node:internal/process/promises:246
          triggerUncaughtException(err, true /* fromPromise */);
          ^
Error: Protocol error (Network.deleteCookies): Invalid parameters name: string value expected
    at /Users/xxxxxx/Downloads/patreon-downloader-master/node_modules/puppeteer/lib/Connection.js:186:56
    at new Promise (<anonymous>)
    at CDPSession.send (/Users/xxxxxx/Downloads/patreon-downloader-master/node_modules/puppeteer/lib/Connection.js:185:12)
    at Page.deleteCookie (/Users/xxxxxx/Downloads/patreon-downloader-master/node_modules/puppeteer/lib/Page.js:361:26)
    at Page.<anonymous> (/Users/xxxxxx/Downloads/patreon-downloader-master/node_modules/puppeteer/lib/helper.js:145:23)
    at Page.setCookie (/Users/xxxxxx/Downloads/patreon-downloader-master/node_modules/puppeteer/lib/Page.js:385:16)
    at Page.<anonymous> (/Users/xxxxxx/Downloads/patreon-downloader-master/node_modules/puppeteer/lib/helper.js:145:23)
    at /Users/xxxxxx/Downloads/patreon-downloader-master/js/cookies.js:5:18
    at Array.forEach (<anonymous>)
    at loadCookies (/Users/xxxxxx/Downloads/patreon-downloader-master/js/cookies.js:4:26)
  -- ASYNC --
    at Page.<anonymous> (/Users/xxxxxx/Downloads/patreon-downloader-master/node_modules/puppeteer/lib/helper.js:144:27)
    at Page.setCookie (/Users/xxxxxx/Downloads/patreon-downloader-master/node_modules/puppeteer/lib/Page.js:385:16)
    at Page.<anonymous> (/Users/xxxxxx/Downloads/patreon-downloader-master/node_modules/puppeteer/lib/helper.js:145:23)
    at /Users/xxxxxx/Downloads/patreon-downloader-master/js/cookies.js:5:18
    at Array.forEach (<anonymous>)
    at loadCookies (/Users/xxxxxx/Downloads/patreon-downloader-master/js/cookies.js:4:26)
    at main (/Users/xxxxxx/Downloads/patreon-downloader-master/bin/patreon-downloader.js:26:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

My cookies.json:

[
	{
		"session_id": "1j-xxxxxx"
	}
]

My env variables:

PATREON_ARTIST: "the_name"
PATREON_ARTIST_NAME: "The Name"
DESTINATION_DIRECTORY: ~/Desktop/the_name
DESTINATION_MACHINE_NAME: ""
DESTINATION_MACHINE_USERNAME: ""
DESTINATION_MACHINE_HOST: ""
KODI_USERNAME: ""
KODI_PASSWORD: ""
KODI_PORT: ""

No downloads detected

Not sure whether it is different for other artists or whether patreon has changed the HTML layout, but the link detection here does not work any more: https://github.com/endofhome/patreon-downloader/blob/master/bin/patreon-downloader.js#L24

Files are not tagged post-file-download unfortunately. I could get it to work by using a[href^="https://www.patreon.com/file"] as selector instead.

cURL call does not follow redirects

curl call here: https://github.com/endofhome/patreon-downloader/blob/master/bin/download-mp3.sh#L17 should have -L set because patreon usually answers with a redirect to the CDN

Where does one set the environment variables?

Hello there!

Apologies for this question, but unfortunately I am unsure where to set the environment variables. The README does not clarify

Sorry to bother and thank you for making this!

Required file downloaded.txt does not exist

JS tries to read persistence/downloaded.txt: https://github.com/endofhome/patreon-downloader/blob/master/js/files.js#L6

After checkout this files does not exist.

cURL call does not set cookies

For many downloads to work, this curl call here: https://github.com/endofhome/patreon-downloader/blob/master/bin/download-mp3.sh#L17 also has to have the session_id cookie set or it will return HTTP 403