pantry-io / pantry

A simple to maintain background job and message queue service. https://nickpoorman.com/2020-07-26-data-pipelines-part-2-retries/

nats-requeue

This is a service that persists NATS messages to disk and retries sending them at a deferred time.

THIS IS STILL BEING IMPLEMENTED. IT IS NOT READY FOR USE.

Getting Started

Installing

go get github.com/nickpoorman/nats-requeue

This will retrieve the library and install the retry-queue command line utility into your $GOBIN path.

Configuration

Badger recommends running with GOMAXPROCS=128.

I would recommend setting your soft ulimit to 65535. Google how to do it on your OS.

AWS ECS

Uses

DLQ (dead letter queue)

TODO

Disk-backed Buffer

TODO

How Requeue Works

All queue meta information is kept in memory and synced to disk.

Checkpointing

In memory meta for a queue consists of a Checkpoint. The checkpoint is used for seeking to the a key in Badger. This allows us to iterate directly between the events we are looking for in a time range.

Contributing

This project uses Flatbuffers to serialize messages. To build the protocol/*.fbs files you must have flatc installed and then run the following.

make protocol

Thanks

• NATS for an awesome distributed messaging system.
• This project uses dgraph-io/badger for storing the messages on disk.