vthiery / cpp-statsd-client Goto Github PK

Hi, thanks for the rare cpp statsd client!

We'd like to use the client but my concern is about if the client is thread safe?
if not do you have some suggestions to use the client in multi-threads web services?

At the moment we have 2 modes for StatsdClient:

1. a simple mode in which every stat is sent immediately in a blocking fashion
2. a batching mode in which every stat is sent to a queue, from which a background thread will, on a secondly basis, send the queued stats from the last round. there is also a batch size which allows making use of the multi metric feature

I propose to make this interface a little more customizable by allowing each of these pieces to be configured individually. The specific pieces I'm talking about are:

• when to send?
  • immediately
  • on some timer in a background thread
  • on some number of stats
  • manually when you tell it to
• what batch size to use?
  • this isnt useful when immediately sending but when using a background thread or when deciding when to send on your own it is useful

Basically I'm hoping to refactor the configuration to allow more control by the end application on how and when the bytes get shipped over the socket.

In my specific case I have an application which does a bunch of heavy processing in a tight loop, when its done it returns the result to a client who requested it and then does some cleanup. This cleanup is done outside of the request loop, ie it returns the result before it blocks on cleaning up so that the requester isnt waiting on cleanup. This cleanup makes an idea spot for me to fire off any stats i've collected during the processing of that request. I could make use of the batchign as-is but i dont want:

1. more threads than i already have in this application
2. any synchronization at all within the hot loop of my program

For this reason it would be really useful to add a mode in which I can set a batch size to whatever maximum makes sense on my network. Have a queue to which my hot loop adds stats as it runs. And then have a method on the StatsdClient that allows me to flush that queue in my cleanup after my algorithm has run.

It probably goes without saying but I'm happy to implement this feature so please feel free to assign me to it unless it sounds like something you'd rather not have in the library.

Currently CI does not check for properly linted code. We use clang-format here with the checked in style but we dont validate it on CI which makes for a bunch of work for any reviewer to sift through. To save some work, lets have the computers do it for us and run lint and fail the build if doing so shows any differences with what is checked in.

Also differences in versions of clang-format will do different things with your style file. Should we pin the version to avoid that problem?

Recently we added github actions to the repo which build and run a test application that publishes stats a few different ways. At the moment all it does is test that the code compiles and doesn't complain when you run it. This is simply not thorough enough. For proper testing we need to:

1. enable linux github action and set it as required (@vthiery can you help me out here?)
2. run statsd with either the graphite or console backend to collect stats from our test program
3. after the test program finishes verify that the stats actually landed at the backend

There may be some interesting (ie more difficult) stuff to test here regarding randomness and the frequency of collection, but I'm sure we can figure something out!

@vthiery i suppose you cannot assign this task to me but suffice it to say i will PR this work

I thought you might want to change the c++17 tag on the repo to reduce possible confusion:

does a cpp-statsd-server exist? thanks

At the moment the API requires you to specify your metric prefix with the trailing . OR to specify your metrix key with a leading . to separate the two in the outgoing metric. We should do this convenience for people and document that it is being done. More info here: #19 (comment)

Would it be much work to add this lib to the public conan repo?

This issue is based on the review comment here: #19 (comment)

The idea here is that we'd like to make it impossible for the client to send messages with bogus formatting but the current API allows users to send metrics manually by specifying the metric type as a string. This opens the door for users to pass a string that is not valid and would force us to validate it and set an error message if invalid. With a small (but breaking) API change we can side step the whole issue by:

1. making the send method private
2. add support for all the valid metric types (set and histogram are missing)

To follow with semver we will need to increment the major revision though.

@vthiery In general do you think this is a worthwhile thing to do?

See https://github.com/vthiery/cpp-statsd-client/blob/v2.0.0/CMakeLists.txt#L4

We were wondering if tags are supported by this client library and how they should be used.
We are looking something similar to python statsd client metrics tagging feature

    import statsd
    c = statsd.StatsClient('localhost', 8125, prefix='foo')
    c.incr('bar', tags = {"key1": "value1", "key2":"value2"})

As discussed here https://stackoverflow.com/questions/49852654/sending-statsd-metrics-with-tags-from-python-client

I have tried to use this lib in an existing CMake proejct.

Canoot get it to work.

Have tried both add_subdirectory and find_package approach.

Going down the find_package route:
No matter what I do it seems the top-level project can't find the installed headers. It seems that INTERFACE_INCLUDE_DIRECTORIES defined by the cpp-stasd-client in the generated /home/fpan/libdir/share/cpp-statsd-client/cmake/cpp-statsd-clientTargets.cmake.

I see this PR has seemingly adressed the issue: #25

Any tips or pointers?

At the moment the cmake setup we have works to build the test program but there are a couple of minor issues with it:

1. the test program doesnt depend on the library (because there is no library target) and so you can get in a situation where you make changes to the header and the test program isnt able to tell it needs to be rebuilt
2. depending on how someone wants to use this library in a downstream project they may need to do some tricks. we should create a library target and mark it as an interface so that downstream users via cmake will know where to look for includes (this includes the test executable as well)
3. we dont set a version or a description
4. other little stuff?

At the moment the documentation about batching and frequency rates is very poor. It would be nice to:

• describe the mechanismes
• briefly explain the implementation and limitations
• give examples

Looking at the batching code I noticed m_mustExit being a "plain" bool, which means if I understand the C++ memory model correctly that the compiler/cpu are not required to provide visibility of the changes to other threads so the background batching thread may spin indefinitely.

suggestion: switch m_mustExit to std::atomic_bool and use memory_order_acq_rel on load()/store() [2]

[1] https://github.com/vthiery/cpp-statsd-client/blob/master/include/cpp-statsd-client/UDPSender.hpp#L76
[2] https://en.cppreference.com/w/cpp/atomic/memory_order#Release-Acquire_ordering