GithubHelp home page GithubHelp logo

benhoyt / graphyte Goto Github PK

View Code? Open in Web Editor NEW
81.0 7.0 19.0 60 KB

Python 3 compatible library to send data to a Graphite metrics server (Carbon)

License: MIT License

Python 100.00%
python graphite carbon metrics async

graphyte's Introduction

graphyte

graphyte on PyPI (Python Package Index) GitHub Actions Tests

graphyte is a small Python library that sends data to a Graphite metrics server (Carbon). We wrote it because the existing graphitesend library didn’t support Python 3, and it also required gevent for asyncronous use. graphyte is tested on Python 3.5+ as well as Python 2.7, and uses the standard library’s threading module for asynchronous use.

The library is on the Python Package Index (PyPI), so to install it, fire up a command prompt, activate your virtualenv if you’re using one, and type:

pip install graphyte

Using graphyte is simple – just call init() to initialize the default sender and then send() to send a message. For example, to send system.sync.foo.bar 42 {timestamp}\n to graphite.example.com:2003 synchronously:

import graphyte
graphyte.init('graphite.example.com', prefix='system.sync')
graphyte.send('foo.bar', 42)

If you want to send asynchronously on a background thread (for example, in a web server context), just specify a send interval. For example, this will setup a background thread to send every 10 seconds:

graphyte.init('graphite.example.com', prefix='system.async', interval=10)
graphyte.send('foo.bar', 42)

If you want to send tagged metrics, the usage is as follows:

graphite.send('foo.bar', 42, tags={'ding': 'dong'})

For more advanced usage, for example if you want to send to multiple servers or if you want to subclass Sender, you can instantiate instances of Sender directly. For example, to instantiate two senders sending to different servers (one synchronous, one using a background thread with send interval 10 seconds), use something like the following:

sender1 = graphyte.Sender('graphite1.example.com', prefix='system.one')
sender2 = graphyte.Sender('graphite2.example.com', prefix='system.two', interval=10)
sender1.send('foo.bar1', 42)
sender2.send('foo.bar2', 43)

If you want to send via UDP instead of TCP, just add protocol='udp' to the init() or Sender() call.

Or, to customize how messages are logged or sent to the socket, subclass Sender and override send_message (or even send_socket if you want to override logging and exception handling):

class CustomSender(graphyte.Sender):
    def send_message(self, message):
        print('Sending bytes in some custom way: {!r}'.format(message))

By default, exceptions that occur when sending a message are logged. If you want to raise and propagate exceptions instead, instantiate Sender with raise_send_errors=True. It's an error to set raise_send_errors when interval is specified.

Socket sending errors are logged using the Python logging system (using logger name “graphyte”). If the sender is initialized with log_sends=True, all sends are logged at the INFO level.

You can also use graphyte to send metrics directly from the command line:

python -m graphyte foo.bar 42

There are command line arguments to specify the server and port and other configuration. Type python -m graphyte --help for help.

Read the code in graphyte.py for more details – it’s pretty small!

graphyte was written by Ben Hoyt and is licensed with a permissive MIT license (see LICENSE.txt).

Related work: delphid has a fork of graphyte which supports the statsd protocol. See the changes on delphid's branch.

graphyte's People

Contributors

barrycarey avatar benhoyt avatar ezequielramos avatar ihoegen avatar plz avatar serphentas avatar vchimishuk avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

graphyte's Issues

Batch and interval arguments are confusing

Hi,

I've been digging into the code and tweaking behaviour as the module wasn't working for me with large metric counts.

My assumption reading the documentation, and standard queuing behaviour, would be that batch size and interval are to ensure steady throughput to a graphite server. However that doesn't seem to be the case. The batch size works in that is how many metrics are sent in one socket operation, but there is no delay based on the interval.

For instance:

  • Create an async sender queue with batch size 50, and interval 5
  • Put 1000 metrics onto the queue
  • I would now expect the lib to send 50 metrics, every 5 seconds, so 1200 message a minute.
  • Instead, I can log graphyte sending 50 messages to the socket at a time, without a delay
2021-07-20 13:22:15,309 - graphite-buffer - INFO - sent message of 6185B, 50 metrics, to graphite:2003 in 0.222 seconds
2021-07-20 13:22:15,536 - graphite-buffer - INFO - sent message of 6204B, 50 metrics, to graphite:2003 in 0.227 seconds
2021-07-20 13:22:15,762 - graphite-buffer - INFO - sent message of 6173B, 50 metrics, to graphite:2003 in 0.226 seconds
2021-07-20 13:22:15,985 - graphite-buffer - INFO - sent message of 6206B, 50 metrics, to graphite:2003 in 0.223 seconds
2021-07-20 13:22:16,211 - graphite-buffer - INFO - sent message of 6205B, 50 metrics, to graphite:2003 in 0.225 seconds

The guilty code being:

                if current_time - last_check_time >= self.interval:
                    last_check_time = current_time
                    for i in range(0, len(messages), self.batch_size):
                       batch = messages[i:i + self.batch_size]
                       self.send_socket(b''.join(batch))
                    messages = []

It seems the algorithm is to check the queue every interval seconds, then flush the entire queue with batch number of metrics in each write operation?

So, is this behaviour intentional, and interval is how often the thread queue basically wakes up and flushes the queue? Or is this buggy behaviour, and it should instead read&send a specific number of metrics (batch #) off the queue each interval?

installing in windows causing error

Logs when installing the graphyte 1.2

Collecting graphyte
  Using cached graphyte-1.2.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\NECXFA~1\AppData\Local\Temp\pip-build-41uqbbiy\graphyte\setup.py", line 21, in <module>
        long_description = f.read()
      File "c:\program files\python35\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2463: character maps to <undefined>

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\NECXFA~1\AppData\Local\Temp\pip-build-41uqbbiy\graphyte\

May be this is because the "" in the README.rst file

Sending large number of messages via async method causes loss of data

When sending 10,000 messages in one burst, we found that many of them didn't get successfully transferred to the graphite relay.

Moving to sync worked, but uses a huge number of socket connections. Adding a max message count or size to the deferred thread queue would be super helpful, as would sending a single message as a pickle format.

Add support for sending data in pickle format

The pickle protocol is somewhat more efficient, especially when sending a bunch of metrics at once. Would be nice to add an option to send data using the pickle format, as per http://graphite.readthedocs.io/en/latest/feeding-carbon.html:

list_of_metric_tuples = [(path, (timestamp, value)), ...]
payload = pickle.dumps(list_of_metric_tuples, protocol=2)
header = struct.pack("!L", len(payload))
message = header + payload

Perhaps add a protocol='pickle' option to Sender.__init__ (defaults to plaintext). It'd be nice if the port argument defaulted to None, and then if None we'd select the default port based on protocol (plaintext=2003, pickle=2004).

If it just sent one at a time in foreground (interval=None) mode, I think that's fine, and the benefits would only kick in if you were in background (interval) mode. But I think that's fine.

Queued messages should not get lost on exception during sending

When I run graphyte in interval mode to let it send messages to Graphite in a background thread, all messages stored until the next scheduled send operation are stored in a queue. Once the next scheduled send is reached, messages will be popped from the queue and batched for sending. However, if the send operation fails (e.g. due to a temporary network outage), this batch of messages will be lost because they will not be put back into the queue for the next try.

I think it would be useful to put messages back into the queue and try them again on the next scheduled interval, together with the ones which might have been added on top of them in the meantime.

If there are use cases where this behaviour is undesired, it could be made configurable in the constructor of the Sender class (retry_failed_messages=True or similar).

Propagate Connection Exceptions

I feel like it would be useful to propagate connection exceptions instead of catching them internally. With the current implementation if send_socket() fails client code has no way of knowing.

Currently I'm required to subclass Sender and override send_socker() to allow my client code to detect a failed connection.

add support for a statsd styled message

Hi, benhoyt, just be utilizing this awesome package for sending metrics to a statsite server. And did some customization in order to comply with the statsite/ statsd protocol.

  • the default graphite protocol: {metric} {value} {timestamp}
  • the statsd styled protocol: {metric}:{value}|{type}

Just in case if anyone else would need this feature, would you like to check out this modification:
the forked branch

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.