leogallego / ansible-stats-crawler

Primary indexer for the raw data used in the other stats reports and dashboards

Stats Crawler

Primary indexer for the raw data used in the other stats reports and dashboards

A lot of the data we used is available via APIs with rate limits (such as GitHub, Meetup, etc). It's preferrable to have that data locally, so we can query it as often as we like. This project handles the import of the primary (upstream) data sources.

The apps that depend on this data will be responsible for the creation of the secondary data themselves - to put it all here would likely lead to many many software dependencies and difficulty in getting it all to play nice.

Basic Outline

There are 3 main directories:

config

This directory contains the necessary input data for the vaious jobs to function. In some cases these will be example files that are modified on the production server, because they contain API keys, etc.

lib

This directory holds the necessary functions and files for the jobs to run. Most of the complexity of each job is to be found here.

tasks

This holds a set of scripts which are to be executed by cron, and should be vaguely readable even to those not familiar with the project. They will call high-level functions defined in lib for each step of the task, so that the flow of the job can be understood.

Deployment

Currently there are two tasks (GitHub and Meetup), and they need to be set up in cron manually. Run it with Rscript path/to/task.R, but be aware that the project uses renv and will call renv::restore at the start of each run to ensure it matches the lockfile.

Eventual plan is to have all the tasks represented in a simple Ansible playbook which defines a system user and entries in that users crontab - each new task would need a matching stanza in the playbook. This is still TBD.

Contribution

All contribution is welcome, please open a PR

License

GPL3