GithubHelp home page GithubHelp logo

amor71 / mnqueues Goto Github PK

View Code? Open in Web Editor NEW
5.0 4.0 0.0 458 KB

Monitored Multiprocessing Queues

Home Page: https://github.com/amor71/mnqueues

License: MIT License

Python 100.00%
python multiprocessing monitoring queue scale python3 python-3 multithreading tracing monitor scaleable python-library queueing queue-manager queue-tasks

mnqueues's Introduction

CircleCI Updates Python 3 Sourcery

What are mnqueues?

mnqueues stands for Monitored Queues - a coupling between Python multiprocessing Queue() and a Monitor entity. A Monitor collect and alerts on Queue usage statistics.

Tracked measures

mnqueues tracks several measures per queue:

  1. Average number of writes to queue per minute,
  2. Average number of reads to queue per minute,
  3. Time spend in queue (latency) in milliseconds.

Installation

To install mnqueues type:

pip install mnqueues

How-To use

To create a monitored queue:

import mnqueues as mnq
from mnqueues.gcp_monitor import GCPMonitor

q = mnq.MNQueue(monitor=GCPMonitor("some-unique-name"))

The MNQueue() object encapsulated Python multiprocessing.Queue() and supports same functions. The MNQueue() object can be passed between processes, like a Queue() object.

Monitors

File Logger

from mnqueues.log_monitor import LOGMonitor

monitor = LOGMonitor("log-file-name")

Log all put() and get() calls to a log file with the the name log-file-name.log with the following format:

[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 5003
[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 4997

Google Cloud Monitor (using StackDriver)

from mnqueues.gcp_monitor import GCPMonitor

monitor = GCPMonitor("unique-name")

All calls to put() and get() are sent to Google Cloud Monitor. The Monitor class sends data to two custom measures:

  1. OpenCensus/mnqueues.{name}.number_queue_get (line, no aggregation on GCP required)
  2. OpenCensus/mnqueues.{name}.number_queue_put (line, no aggregation on GCP required)
  3. OpenCensus/mnqueues.{name}.time_in_queue_distribution (heat-map with sum, shows latency distribution)

Note that {name} is passed as a parameter when constructing the Monitor and it aims to assist in creating dash-boards for specific use-cases.

See for details.

Examples

GCP

  1. View Google Cloud (GCP) Monitoring dashboard showing queue.put() and queue.get() rates per second, generated by running pytest on the project tests folder.
  2. Monitoring queues with real-time web-socket trading data for LiuAlgoTrader.

Further examples

Can be found in the tests folder.

Contributing

Contributions are highly appreciated. Please review our Code of Conduct. Bug reports & feature requests can be left in the Issues section, or email me at [email protected]

mnqueues's People

Contributors

amor71 avatar pyup-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

mnqueues's Issues

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.