Internetarchive-YouTube

🚀 GitHub Action and CLI tool to archive YouTube channels by automatically uploading an entire YouTube channel to archive.org in few clicks.

📌 Global Requirements

All you need is an Internet Archive account.

🔧 Usage

⚡️ To use this tool as a GitHub Action, jump to GitHub Action: Getting Started.
🧑‍💻 To use this tool as a command line interface (CLI), jump to CLI: Getting Started.

⚡️ GitHub Action: Getting Started

Using internetarchive-youtube as a GitHub Action instructions

Fork this repository.
Enable the workflows in your fork.

Create a backend database (or JSON bin).
Add your Archive.org credentials to the repository's Actions secrets:

ARCHIVE_USER_EMAIL
ARCHIVE_PASSWORD

Add a list of the channels you want to archive as a CHANNELS secret to the repository's Actions secrets:

The CHANNELS secret should be formatted like this example:

CHANNEL_NAME: CHANNEL_URL
FOO: FOO_CHANNEL_URL
FOOBAR: FOOBAR_CHANNEL_URL
SOME_CHANNEL: SOME_CHANNEL_URL

Don't add any quotes around the name or the URL, and make sure to keep one space between the colon and the URL.

Add the database secret(s) to the repository's Actions secrets:

If you picked option 1 (MongoDB), add this secret:

MONGODB_CONNECTION_STRING The value of the secret is the database conneciton string.

If you picked option 2 (JSON bin), add this additional secret:

JSONBIN_KEY
The value of this secret is the MASTER KEY token you copied from JSONbin.

Run the workflow under Actions manually, or wait for it to run automatically every 6 hours.

That's it!

🧑‍💻 CLI: Getting Started

Using internetarchive-youtube as a CLI tool instructions

Requirements:

🐍 Python>=3.7

⬇️ Installation:

pip install internetarchive-youtube

# Install and login to internetarchive
pip install internetarchive
ia configure

🗃️ Backend database:

Create a backend database (or JSON bin) to track the download/upload overall progress.
If you choose MongoDB, export the connection string as an environment variable:

export MONGODB_CONNECTION_STRING=mongodb://username:password@host:port

# or add it to your shell configuration file:
echo "MONGODB_CONNECTION_STRING=$MONGODB_CONNECTION_STRING" >> "$HOME/.$(basename $SHELL)rc"
source "$HOME/.$(basename $SHELL)rc"

If you choose JSONBin, export the master key as an environment variable:

export JSONBIN_KEY=xxxxxxxxxxxxxxxxx

# or add it to your shell configuration file:
echo "JSONBIN_KEY=$JSONBIN_KEY" >> "$HOME/.$(basename $SHELL)rc"
source "$HOME/.$(basename $SHELL)rc"

⌨️ Usage:

usage: ia-yt [-h] [-p PRIORITIZE] [-s SKIP_LIST] [-f] [-t TIMEOUT] [-n] [-a] [-c CHANNELS_FILE] [-S] [-C]

optional arguments:
  -h, --help            show this help message and exit
  -p PRIORITIZE, --prioritize PRIORITIZE
                        Comma-separated list of channel names to prioritize
                        when processing videos
  -s SKIP_LIST, --skip-list SKIP_LIST
                        Comma-separated list of channel names to skip
  -f, --force-refresh   Refresh the database after every video (Can slow down
                        the workflow significantly, but is useful when running
                        multiple concurrent jobs)
  -t TIMEOUT, --timeout TIMEOUT
                        Kill the job after n hours (default: 5.5)
  -n, --no-logs         Don't print any log messages
  -a, --add-channel     Add a channel interactively to the list of channels to
                        archive
  -c CHANNELS_FILE, --channels-file CHANNELS_FILE
                        Path to the channels list file to use if the
                        environment variable `CHANNELS` is not set (default:
                        ~/.yt_channels.txt)
  -S, --show-channels   Show the list of channels in the channels file
  -C, --create-collection
                        Creates/appends to the backend database from the
                        channels list

🏗️ Creating A Backend Database

Creating A Backend Database instructions

Option 1: MongoDB (recommended).
- Self-hosted (see: Alyetama/quick-MongoDB or dockerhub image).
- Free cloud database on Atlas.
Option 2: JSON bin (if you want a quick start).
- Sign up to JSONBin here.
- Click on VIEW MASTER KEY, then copy the key.

📝 Notes

Information about the MONGODB_CONNECTION_STRING can be found here.
Jobs can run for a maximum of 6 hours, so if you're archiving a large channel, the job might die, but it will resume in a new job when it's scheduled to run.
Instead of raw text, you can pass a file path or a file URL with a list of channels formatted as CHANNEL_NAME: CHANNEL_URL or in JSON format {"CHANNEL_NAME": "CHANNEL_URL"}.

slickag / internetarchive-youtube Goto Github PK