jwilner / mongodb-shovel

a simple shovel for republishing events from mongodb capped collections to message brokers

mongodb-shovel

A simple shovel for republishing events from MongoDB capped collections to RabbitMQ for dispatch

what?

MongoDB's oplog (and capped collections more generally) is ripe for driving an event bus. The idea behind this shovel is to do one simple thing -- tail a capped collection and shovel matching results into a message broker (I'll do RabbitMQ first).

In order to provide basic ordering guarantees through to the broker, this will eschew any real concurrency; multiple shovels could be run simultaneously to achieve concurrency while still preserving the ordering guarantees for the individual queries.

Local file system state will be used to track the id of the last record read.