spedrickson / ytchat-backend

nestJS backend for ytchat mod tools

About

Backend API service for YouTube chat history written in NestJS. Requires a configured MongoDB instance to connect to.

Swagger documents will also be generated automatically when running.

This was started as a practice project to learn NestJS and JS/TS in general. Some things about it may strange or incorrect, feel free to point out any issues.

Limitations

Currently, this supports one Youtube channel per active process, determined by the YTCHAT_CHANNELID environment variable.

This means supporting multiple channels will require multiple containers each with their own port.

Also, the swagger documentation is currently auto-generated and needs to be filled out more clearly.

Installation

# from the source directory
$ npm install

Configuration

Before launching, set any necessary environment variables.

# bash
YTCHAT_CHANNELID=...

# powershell
$env:YTCHAT_CHANNELID = "..."

Supported variables

Variable	Purpose	Default
YTCHAT_CHANNELID	Which MongoDB collection should be queried.	UCrPseYLGpNygVi34QpGNqpA (Ludwig)
YTCHAT_BACKEND_PORT	Which port the backend API should listen on.	3000
YTCHAT_BACKEND_MONGOSTRING	The MongoDB Connection String. The DB is selected with the above YTCHAT_CHANNELID env variable.	'mongodb://user:[email protected]:27017'

API Keys/Authentication

The backend uses API keys with associated permissions for authentication. Each channel DB should have a collection called users that stores these API keys/permisisons. There's currently no built-in method for generating keys/permissions, so they will have to be inserted into the DB manually.

An example user (in JSON format) is as follows:

{
  "name": "user's name",
  "apikey": "any_unique_string",
  "isAdmin": false,
  "perms": {
    "view": true,
    "comment": false,
  }
}

Permissions:

isAdmin: Should override all other permissions, and allow full access to viewing/commenting/etc.
view: Can read chat history for users/perform queries on the filtered page.
comment: Can leave mod comments for users.

Running the app

# development
$ npm run start

# watch mode
$ npm run start:dev

# production mode
$ npm run start:prod

Testing

This was started as a personal practice project, so I didn't write any unit tests even though I definitely should have.

NestJS will generate some boilerplate unit tests if you'd like to add some, feel free to submit a pull request if so!

Support

Feel free to submit any issues or pull requests. No warranty or support is guaranteed beyond that.

License

This code is under the MIT license.

ytchat-backend's Issues

Store username quicksearch data

Outline

Currently there is a scalability problem with the fillAuthorCache() function. As-is, it asks MongoDB to perform the following aggregation:

1. sort all messages by timestamp (should already use index)
2. group messages by channelID with the latest message's userdata and timestamp as extra fields
3. return collection of every unique user to backend,

This means that it's using O(n) space complexity, with N as the number of unique users that have left a message since ingest began. And if the MongoDB host doesn't have enough memory, it'll become IO-bottlenecked by writing partial results to disk.

This also means the backend is using O(n) space to store this information, which may not be best practices.

Currently takes ~3 minutes to run against 1.1m users, and will continue to grow from there.

Proposed Solution

• Change the fillAuthorCache() function to a fillAuthorCollection() function, which uses aggregation to store the author data in the DB as a separate collection named authors.
• Change getAuthorBySearch() endpoint function to search database instead of local authorCache. Will need to investigate possible solutions to maintain fuzzy search, or change it to simple text matching and/or regex.
• Change the refreshAuthorCache() function to update the authors collection, rather than pulling new authors into the backend's authorCache.
• Remove authorCache entirely, now that it is unused.

Support multiple YouTube channels per backend instance

Currently ytchat-backend only supports a single YouTube channel per instance, configured through the YTCHAT_CHANNELID environment variable.

Instead, it should be refactored to accept channelID as a query parameter.

Steps needed for this enhancement:

• Remove YTCHAT_CHANNELID env variable
• Refactor Mongoose to provide multiple mongoDB connections (example 1, example 2). For scalability, consider the Flyweight design pattern.
• Refactor API permissions to be at the channel level, rather than the backend instance level. Easiest way would be to add a function to src/user/schemas/user.service.ts that accepts a channelID and API key, and returns the auth details.
• Update ytchat-frontend to use new query parameter.