This is a rss-feed cloning service designed to download data from rss feeds and upload to multiple cloud services such as google-drive and telegram etc. (more to come). The service is able to do simuntaneous downloads and uploads, It also supports displaying the live progress status of transfer queue on telegram. A database backend is also provided for storing the state of the content.

The service incorporates a scalable structure designed to be easily extensible over time as needed. The four main parts called rss-parser, download-manager, upload-manager and status-manager communicate with each other via channels.

The rss parser is powered by gofeed. The parser runs in a seperate goroutine and there can be as many parsers hooked up to the number of urls. It periodically keeps on pinging the rss feed url to get new feeds. If new feed items are found then they are registered in the database repository and sent to downloader manager via the download channel.

The download manager is powered by anacrolix/torrent. The download manager also runs in a seperate goroutine listening for urls to download over the download channel that are provided by the rss-parser. the download is carried out by the download manager and a listener is hooked up to the download for listening events. This listener is also responsible for forwarding the path of the downloaded file to the upload-manager.

The upload manager handles all the uploading in the service. There are two uploaders provided: telegram and google-drive. The upload manager listens for completed downloads on the upload channel in a seperate goroutine. The completed upload results are then forwarded to a upload completion channel where a goroutine cleans up the file and sends a notification on telegram.

This component is responsible for creating progress status messages and displaying in a telegram chat. A status loop is started in a goroutine when seasonal-go fires up. There can only be one status message per chat so when the /status command is sent to bot, the previous status message is deleted and a new one is sent. This status loop updates the progress status messages on telegram periodically with intervals to avoid telegram rate limiting. Status manager also provides a bridge to telegram api methods such as SendMessage, EditMesssage and DeleteMessage.

The configuration is handled via config.json file.

{
  "bot_token": "",
  "download_dir": "temp",
  "tg_app_hash": "",
  "tg_app_id": 0,
  "telegram_chat_id": "",
  "telegram_channel_id":  0,
  "sa_dir": "accounts",
  "telegram_upload": false,
  "google_drive_upload": true,
  "google_drive_folder_id": "",
  "database_type": "memory",
  "database_url": "",
  "concurrent_transfers" : 2,
  "rss_urls": [
    ""
  ]
}

A sample-config.json file is also provided in the repo.

There are three backends provided at the moment: memory, mongodb, bitcask.

mongodb: database_url field in config is used to set the mongodb server url

bitcask: database_url field in config is used to set bitcask folder name, this field expects a path on local filesystem