spedrickson / ytchat-ingest

Ingest messages from YouTube live chat to MongoDB

ytchat-ingest

Ingest messages from YouTube live chat to MongoDB.

This is related to the projects ytchat-backend and ytchat-frontend.

Limitations

Currently, this program is designed to ingest from only one chat at a time.

Additionally, there is no good way to find pending YouTube live streams, so it checks for a redirect from https://youtube.com/CHANNEL_ID/live. This means that it only supports one live stream per channel at a time, depending on which the URL redirects to.

Setup

Install dependencies with:

pip install -r requirements.txt

Configuration

Before launching, set any necessary environment variables.

# bash
YTCHAT_CHANNELID=...

# powershell
$env:YTCHAT_CHANNELID = "..."

Supported variables

Variable	Purpose	Default
YTCHAT_CHANNELID	Which MongoDB collection messages should be inserted into.	UCrPseYLGpNygVi34QpGNqpA (Ludwig)
YTCHAT_INGEST_MONGO_URL	Host URL for MongoDB instance that should store messages.	localhost
YTCHAT_INGEST_MONGO_PORT	Port for the MongoDB instance that should store messages.	27017

MongoDB Setup

Note: you will need to set an index that enforces uniqueness of messages based on the id field (not _id). Otherwise, you will occasionally get duplicate messages inserted into the DB.

Running

python3 main.py CHANNEL_ID

License

This program is released under the MIT License

ytchat-ingest's Issues

remove playwright dependency

Currently the ingest tool checks if a channel is live by going to
https://youtube.com/CHANNEL_ID/live with a Playwright instance.

This is very inefficient, and can be done with a simple GET request, then checking the href attribute of the <link rel="canonical" /> tag.

Example of how it has be done in node.

Support YouTube handles

Now that YouTube supports @handles for accounts, it would be valuable for the ytchat-ingest tool to support them as well.

Current impact of using a handle with the tool:
Currently the only downside of launching the tool with a @handle instead of a channelID is messages will be put in a collection named after the handle rather than the channelID.

Workaround:
Just launch the tool with a channelID (UC...) instead of a @handle

Long-term fix:
For consistency, the tool should resolve @handles into channelIDs on launch, and use that as the collection name. This should also be more stable against end-user changes like new handles, since channelIDs are unchangeable.