buzzfeed-openlab / rss-puppy

A watchdog tool for monitoring RSS feeds

RSS Puppy

RSS Puppy is an BuzzFeed Open Lab project from our first cohort. You're welcome to fork it and build on it and learn from it, but unfortunately we're not able to provide support. 😢

A watchdog tool for monitoring RSS feeds

This tool is designed to monitor RSS feeds in bulk, and to generate machine friendly notifications when new entries appear. While there exists no shortage of RSS readers and web-based notification services, nothing we found combines easy managment of hundreds of RSS feeds with the flexibility to direct output to a variety of data stores or over disparate protocols.

This monitor can be run on any cloud service provider, and requires only Nodejs and a PostgreSQL database. Also, it is trivial to add output handlers which can pipe feed entry data to any service you use.

Read more about the motivation and design.

How to run

Get the code

• git clone https://github.com/buzzfeed-openlab/rss-puppy.git
• cd rss-puppy; npm install
• cp ./sample-config.json ./config.json

Set up a database

The monitor uses a PostgreSQL database to keep track of feeds and entries.

An easy way to get a reliable, automatically backed up database is to use AWS. Log into the management console, navigate to the RDS dashboard, click Launch a DB instance, select PostgreSQL, and follow the rest of the configuration steps.

Once you have your DB, you can tell the monitor about it using the config.json file:

{
    "dbconfig": {
        "user": "DB USER",
        "password": "DB USER PW",
        "url": "POSTGRES DB PATH",
        "port": 5432,
        "dbname": "DB NAME",
        "initScript": "./monitor/init-feed-db.sql"
    },
    ...
}

Configure your feeds

In the config file there will be a section called "feeds" and a section called "throttling".

{
    "feeds": [
        "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0001440512&type=&dateb=&owner=exclude&start=0&count=40&output=atom",
        ...
    ],

    "throttling": {
        "monitorFrequency": 8000,
        "maxConcurrent": 10,
        "concurrentInterval": 1000
    },
    ...
}

• "feeds" is an array of RSS feed urls that will be monitored.
• "throttling" is broken into several parts
  • "monitorFrequency": How often the monitor will check to see if it needs to query any "old" RSS feeds (ie: ones that haven't been queried in awhile).
  • "maxConcurrent": The maximum number of concurrent queries the monitor will make (excess queries will be queued).
  • "concurrentInterval": The interval to wait between making "maxConcurrent" queries (ie: X queries per 10 seconds, or X queries per 60 seconds).

Configure your outputs

Outputs are modules of code that listen for events that the monitor emits and do something useful with the resulting data.

There are several different kinds of events:

• "new-entry": Emitted when the monitor encounters an entry that it has not seen before. Handlers will be invoked with the entry as a json object, and the feed url as a string.
• "checking-old-feeds": Emitted whenever the monitor wakes up to look for feeds to query (approx every "monitorFrequency" seconds).
• "old-feed": Emitted whenever the monitor finds a feed that hasn't been queried in awhile and needs to be checked. Handlers will be invoked with the feed url as a string.
• "entry": Emitted whenever an entry is parsed from a feed. Note that feeds will be queried and parsed over and over again, so this will be emitted for the same entry many times. Handlers will be called with the entry as a json object and the feed url as a string.

{
	"outputs": [
        {
            "file": "./outputs/debug-logger.js",
            "config": {
                "showOldFeedMessages": true
            }
        },
        ...
    ],
    ...
}

Run the monitor!

node ./run.js [/path/to/config.json]

rss-puppy's Issues

make "old feed" time configurable

containerize

If this had a docker file it would be even easier to throw it up into a cloud provider

On error event

Hi. It's me again. Can we have some kind of on-error event added please? It'd be nice if it also passed some clear text about error.

use a timequeue to help with rate limiting

SQL

From a look at it - there aren't any Postgres specific SQL commands on there - is it just mentioned as a requirement as it's your SQL server of choice - or am I missing something?

add cli options

specifically, it would be nice to support either wiping out the "entries" table or leaving it alone on startup

Error : ENOTFOUND

Hello. I've been trying to set RSS puppy up. I'm getting { [Error: getaddrinfo ENOTFOUND] code: 'ENOTFOUND', errno: 'ENOTFOUND', syscall: 'getaddrinfo' } and I have no idea what's referring to. Here's config file:
{
"feeds": [ "https://globenewswire.com/Rss/search/XwXK98ZDWLc8N4m3FssORcAcOHv2hsB6FxDQQN3Yc3oAyuPrMfAte0Pc9aqo_dXL"
],

"throttling": {
    "monitorFrequency": 8000,
    "oldFeedThreshold": 30,
    "maxConcurrent": 10,
    "concurrentInterval": 1000
},

"dbconfig": {
    "user": "rsspuppy",
    "password": "pwhere",
    "url": "127.0.0.1",
    "port": 5432,
    "dbname": "rsspuppy",
    "initScript": "./monitor/init-feed-db.sql"
},

"outputs": [
    {
        "file": "./outputs/debug-logger.js",
        "config": {
            "showOldFeedMessages": true
        }
    }
],

"exitOnError": true

}

Also, it'd be nice if there was more documentation available