logrhythm / queuenado

Implementation of fast queue patterns (Push/Pull, request/reply, pub/sub, dealer/router) for C++

QueueNado

QueueNado is a collection of C++ implemntations of ZMQ fast queue patterns (Push/Pull, request/reply, pub/sub, dealer/router) for C++

Requirements

1. g3log
2. g3sinks
3. FileIO
4. ZMQ v3.2.5
5. CZMQ v3.0.2
6. ZLIB Most Linux distributions carry zlib. Example: In Ubuntu it can be installed with sudo apt-get install zlib1g-dev
7. StopWatch
8. DeathKnell
9. boost For installing boost please follow their instructions.

Build and Install

Example of installation

cd Queuenado
cd 3rdparty
unzip gtest-1.7.0.zip
cd ..
mkdir build
cd build
cmake ..
make -j

Executing the unit tests

sudo ./UnitTestRunneer

Installing

sudo make install

Alternative on Debian systems

make package
sudo dpkg -i Queuenado-<package_version>Linux.deb

Rifle - Vampire

Rifle - Vampire implements the push / pull messaging pattern in zmq. The communication setup supports the following client/server communications

Client -to- Server

• one-to-one
• one-to-many
• many-to-one

In a many scenario, it is the "one" side that does the bind ([Vampire.h] [Rifle.h]) on the queue.

Rifle - Vampire Limitation

It cannot do many-to-many

Use Cases for Rifle - Vampire

• Reliable messaging without responses
• High performance (500k msgs per second or higher) with zero_copy
• Process to process communication

API

[Vampire.h] [Rifle.h]

Test usage

[RifleVampireTests.cpp]

Shotgun - Alien

Shotgun - Alien implements the pub / sub messaging pattern in zmq.

Use Cases for Shotgun - Alien

• One to many: one sender communicating with many listeners.
• Not high performance around 10k msgs a sec. This can be improved by batching many messages together.
• Process to process communication
• All listeners receive every message sent.

Known limitations and issues

• Slow joiner issues don't matter or can be worked around

API

[Shotgun.h] [Alien.h]

Test usage

[ShotgunAlienTests.cpp]

Headcrab - Crowbar

Headcrab - Crowbar implements request / reply messaging pattern in zmq.

Use Cases for Headcrab - Crowbar

• Need to receive acknowledgement from the receiver for each message sent to it
• Know that their will always be a response (otherwise the socket will be broken)

Known limitations and issues

• Lower performance (60k msgs a sec)

[Headcrab.h] [Crowbar.h]

Test usage

[CrowbarHeadcrabTests.cpp]

Harpoon - Kraken

Harpoon - Kraken implements a streaming version of pub / sub. It enables data streaming from a publisher to a subscriber.

Kraken Purpose:

The publisher sending streaming data.

Usage example calls from the API: [kraken.h]

• SendTidalWave() : Send a data chunk to the subscriber ([harpoon]). The call blocks until there is space available in the queue. Returns TIMEOUT, INTERRUPT, CONTINUE status to indicate the status of the underlying queue.

• FinalBreach() : Call to subscriber ([harpoon]) to indicate the end of a stream.

Harpoon: Subscriber that receives the data

Usage example calls from the API:

• Aim() : Set location of the queue (tcp)
• Heave() : Request data and wait for the data to be returned. Returns TIMEOUT, INTERRUPT, VICTORIOUS, CONTINUE to indicate status of the stream. VICTORIOUS means that the stream has completed.

API

[[Kraken.h]] (https://github.com/LogRhythm/QueueNado/blob/master/src/Kraken.h) [KrakenBattle.h] [Harpoon.h] [HarpoonBattle.h]

Test usage

• [KrakenBattleTest.cpp]
• [HarpoonKrakenTests.cpp]
• [KrakenIntegrationTest.cpp]

Notifier - Listener

Notifier - Listener extends the Shotgun - Alien implementation of the pub / sub messaging pattern in zmq.

The Notifier - Listener classes are wrappers around the Shotgun - Alienqueueing framework. A Notifier is used to place a message onto a queue that is read from by multiple Listeners. It uses handshake communications, where each Listener must respond back to the Notifier that it received the message. A Notification is deemed successful only if every expected Listener responds to the Notifier.

Use cases for Notifier/Listener

• One-to-many with handshake feedback
• Alerting multiple processes of an event, or a call to action

Limitations for Notifier - Listener

The same as for Shotgun - Alien

Boomstick - Skeleton

The Boomstick - Skeleton is used for connecting to ElasticSearch over a wrapper. At the moment part of the pattern implementation is not open sourced and still proprietary. Until further notice it is not recommended to use the BoomStick - Skeleton classes.