amor71 / mnqueues

Monitored Multiprocessing Queues

What are mnqueues?

mnqueues stands for Monitored Queues - a coupling between Python multiprocessing Queue() and a Monitor entity. A Monitor collect and alerts on Queue usage statistics.

Tracked measures

mnqueues tracks several measures per queue:

1. Average number of writes to queue per minute,
2. Average number of reads to queue per minute,
3. Time spend in queue (latency) in milliseconds.

Installation

To install mnqueues type:

pip install mnqueues

How-To use

To create a monitored queue:

import mnqueues as mnq
from mnqueues.gcp_monitor import GCPMonitor

q = mnq.MNQueue(monitor=GCPMonitor("some-unique-name"))

The MNQueue() object encapsulated Python multiprocessing.Queue() and supports same functions. The MNQueue() object can be passed between processes, like a Queue() object.

Monitors

File Logger

from mnqueues.log_monitor import LOGMonitor

monitor = LOGMonitor("log-file-name")

Log all put() and get() calls to a log file with the the name log-file-name.log with the following format:

[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 5003
[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 4997

Google Cloud Monitor (using StackDriver)

from mnqueues.gcp_monitor import GCPMonitor

monitor = GCPMonitor("unique-name")

All calls to put() and get() are sent to Google Cloud Monitor. The Monitor class sends data to two custom measures:

1. OpenCensus/mnqueues.{name}.number_queue_get (line, no aggregation on GCP required)
2. OpenCensus/mnqueues.{name}.number_queue_put (line, no aggregation on GCP required)
3. OpenCensus/mnqueues.{name}.time_in_queue_distribution (heat-map with sum, shows latency distribution)

Note that {name} is passed as a parameter when constructing the Monitor and it aims to assist in creating dash-boards for specific use-cases.

See for details.

Examples

GCP

1. View Google Cloud (GCP) Monitoring dashboard showing queue.put() and queue.get() rates per second, generated by running pytest on the project tests folder.
2. Monitoring queues with real-time web-socket trading data for LiuAlgoTrader.

Further examples

Can be found in the tests folder.

Contributing

Contributions are highly appreciated. Please review our Code of Conduct. Bug reports & feature requests can be left in the Issues section, or email me at [email protected]

mnqueues's Issues

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.