datadog / uwsgi-dogstatsd

uwsgi dogstatsd plugin

uwsgi-dogstatsd

uWSGI plugin to emit DogStatsD metrics to Datadog via the Datadog Agent

INSTALL

The plugin is 2.x friendly, so installation can be done directly from the repo:

uwsgi --build-plugin https://github.com/Datadog/uwsgi-dogstatsd

If you are packaging the plugin for distribution, please read the uWSGI Guide for Packagers on plugin placement and extra directives like plugin_dir.

USAGE

Depends on a Datadog Agent to be installed and by default listens for dogstatsd on port 8125.

Configure your .ini file to enable the metrics subsystem, and enable the dogstatsd plugin.

Here's a

[uwsgi]
master = true
processes = 8
threads = 4

http = :9090

# DogStatsD plugin configuration
enable-metrics = true
plugin = dogstatsd
stats-push = dogstatsd:127.0.0.1:8125,myapp

# Application to load
wsgi-file = app.py
...

You can also add additional tags or filter which metrics are published (or how they are published) using one or more optional configuration options:

stats-push = dogstatsd:127.0.0.1:8125,myapp
dogstatsd-extra-tags = app:foo_service,instance:1
dogstatsd-no-workers = true
dogstatsd-all-gauges = true
dogstatsd-whitelist-metric = core.busy_workers
dogstatsd-whitelist-metric = core.idle_workers
dogstatsd-whitelist-metric = core.overloaded
dogstatsd-whitelist-metric = socket.listen_queue

This will begin producing metrics with the prefix defined in the configuration, myapp here:

myapp.core.avg_response_time
myapp.core.busy_workers
myapp.core.idle_workers
myapp.core.overloaded
myapp.core.routed_signals
myapp.core.total_rss
myapp.core.total_tx
myapp.core.total_vsz
myapp.core.unrouted_signals
myapp.rss_size
myapp.socket.listen_queue
myapp.vsz_size
myapp.worker.avg_response_time
myapp.worker.core.exceptions
myapp.worker.core.offloaded_requests
myapp.worker.core.read_errors
myapp.worker.core.requests
myapp.worker.core.routed_requests
myapp.worker.core.static_requests
myapp.worker.core.write_errors
myapp.worker.delta_requests
myapp.worker.failed_requests
myapp.worker.requests
myapp.worker.respawns
myapp.worker.rss_size
myapp.worker.total_tx
myapp.worker.vsz_size

The metrics are tagged and split where there are more than one occurrence, such as CPU core, worker.

Read more on the uWSGI Metrics subsystem for further explanation on metrics provided.

uwsgi-dogstatsd's Issues

How do you tag metrics?

I need to differentiate metrics coming from different environments.

I see some work for tags in the plugin.c file, but I'm not sure how to use it?

Is there an example on using tags?

This plugin makes uwsgi listen on udp 0.0.0.0

When turned on, uwsgi is listening on randomly selected udp port on public interface

You can find netstat output below.

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 0.0.0.0:55138           0.0.0.0:*                           29394/uwsgi     

Is it possible to make this plugin listen on lo interface?

Minor: Readme says dogstatsd-no-workers = true, but output has worker metrics

The example in the README.md says:
...
dogstatsd-no-workers = true
...

This will begin producing metrics with the prefix defined in the configuration, myapp here:
....
myapp.worker.avg_response_time
myapp.worker.core.exceptions
myapp.worker.core.offloaded_requests
myapp.worker.core.read_errors
myapp.worker.core.requests
myapp.worker.core.routed_requests
myapp.worker.core.static_requests
myapp.worker.core.write_errors
myapp.worker.delta_requests
myapp.worker.failed_requests
myapp.worker.requests
myapp.worker.respawns
myapp.worker.rss_size
myapp.worker.total_tx
myapp.worker.vsz_size

But iff dogstatsd-no-workers = true, that means there shouldn't be any worker.*, right?

Correct config for kubernetes

I am trying to run this plugin on Kubernetes. Having followed the dogstatsd setup instructions here, I've then added this to my uwsgi_config.ini:

enable-metrics = true
plugin = dogstatsd
stats-push=dogstatsd:$(DD_AGENT_HOST):8125,uwsgi
print = Pushing uwsgi stats to dogstatsd on $(DD_AGENT_HOST):8125
stats-pusher-default-freq=10
dogstatsd-extra-tags = env:$(AWS_ENVIRONMENT),service:uwsgi

where DD_AGENT_HOST is defined in my deployment.yaml as:

- name: DD_AGENT_HOST
   valueFrom:
       fieldRef:
            fieldPath: status.hostIP

This runs successfully, in the sense that this appears in the logs:

    Pushing uwsgi stats to dogstatsd on 172.19.30.101:8125
    ...
    initialized 54 metrics

But nothing appears in datadog. I also don't see any errors. Is there anything obvious I've missed here? Or anything I can do to debug the config?

per-worker tagged metrics seemingly missing

haven't had time to debug this thoroughly, but given i just stumbled upon this, is this expected working at this point? i've installed the plugin, and some metrics are making it to datadog, but seemingly all the ones that'd involve worker or core tag rollups are missing. i.e. the only ones there are core.busy_workers, core.idle_workers, core.overloaded, core.avg_response_time and a few more. some of them look pretty screwy, too.

just wanna check before i start digging in on this one- thanks for writing it!

Warning logs should be able to be ignored

Get log warnings when there are no metrics to push which happens often if the metrics are pushed continuously . These logs do not seem to be ignorable and are not very helpful and are bringing cost up. These logs should be able to be suppressed.

The warning message:
*** WARNING: Dogstatsd stats pusher configured but there are no metrics to push ***

Writing metrics to UNIX socket.

How much work would it take to make the plugin work with datadog unix sockets?
https://docs.datadoghq.com/developers/dogstatsd/unix_socket/

Metric <app>.worker.respawns aggregrated incorrectly

Inside uwsgi, the backing variable of <app>.worker.respawns is equal to the total number of respawns for the lifetime of the process, but uwsgi-dogstatsd treats it as a statsd type "counter", and the result is as if it was respawning N times per period where N is <num_respawns> * <flushes_per_period> for every period.

This metric in particular should be logged as a statsd gauge, but I suspect more of uwsgi's "counter" types also translate more correctly to statsd gauges than counters.

Unable to load uWSGI plugin when datadog agent is run in separate container

I'm having an issue running the plugin. My setup is the following:

Docker Container 1

• Image: https://hub.docker.com/r/tiangolo/uwsgi-nginx/
• Python 3.5
• DataDog Python library
• App installed at /app

Docker Container 2

• Image: datadog/docker-dd-agent

I have application metrics from Container 1 successfully being sent to dogstatsd in Container 2, and metrics appear in DD.

The plugin appears to install correctly during the Docker build process and I see the following lines confirming it:

Step 13/15 : RUN uwsgi --build-plugin https://github.com/Datadog/uwsgi-dogstatsd
 ---> Running in 7695a0bad109
[uWSGI] getting INI configuration from /app/uwsgi.ini
open("./dogstatsd_plugin.so"): No such file or directory [core/utils.c line 3686]
!!! UNABLE to load uWSGI plugin: ./dogstatsd_plugin.so: cannot open shared object file: No such file or directory !!!
Cloning into 'uwsgi-dogstatsd'...
*** uWSGI building and linking plugin from https://github.com/Datadog/uwsgi-dogstatsd ***
[gcc -pthread] dogstatsd_plugin.so
build time: 0 seconds
*** dogstatsd plugin built and available in dogstatsd_plugin.so ***

When I add the following lines to my uwsgi.ini file:

enable-metrics = true
plugin = dogstatsd
stats-push = dogstatsd:172.17.0.2:8125,mapi

and then rebuild the container and attempt to start it, I receive the following error:

!!! UNABLE to load uWSGI plugin: ./dogstatsd_plugin.so: cannot open shared object file: No such file or directory !!!

[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini

unable to find "dogstatsd" stats_pusher

2017-08-16 13:44:27,629 INFO exited: uwsgi (exit status 1; not expected)

2017-08-16 13:44:30,638 INFO spawned: 'uwsgi' with pid 16

[uWSGI] getting INI configuration from /app/uwsgi.ini

open("./dogstatsd_plugin.so"): No such file or directory [core/utils.c line 3686]

!!! UNABLE to load uWSGI plugin: ./dogstatsd_plugin.so: cannot open shared object file: No such file or directory !!!

[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini

unable to find "dogstatsd" stats_pusher

Am I required to have dogstatsd installed on the same container in which I'm running my uwsgi application in order to use this plugin?

Thank you for your help!

Permission denied on Kubernetes

On EKS, I'm getting this error in the uwsgi logs.

Anyone seen it before and how to remedy?

dogstatsd_send_metric()/sendto(): Permission denied [/uwsgi-dogstatsd/plugin.c line 173]

Thanks!

David

Sample rate option

We are pushing too many metrics to dogstatsd and need the ability to set a sample rate on how often the metrics get sent.

Instructions to deploy specific commit/tag/release

I'd like to add

uwsgi --build-plugin https://github.com/Datadog/uwsgi-dogstatsd

to my CI scripts but I want to freeze the dependency rather than track master.

What's the best way to do this?

Dogstatsd aggregator is complaining about empty values

I am seeing lots of these in the dogstatsd log.

2015-01-14 23:13:40 UTC | ERROR | dd.dogstatsd | dogstatsd(dogstatsd.py:306) | Error receiving datagram
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/dogstatsd.py", line 294, in start
    aggregator_submit(message)
  File "/opt/datadog-agent/agent/aggregator.py", line 512, in submit_packets
    parsed_packets = self.parse_metric_packet(packet)
  File "/opt/datadog-agent/agent/aggregator.py", line 442, in parse_metric_packet
    raise Exception('Metric value must be a number: %s, %s' % (name, raw_value))
Exception: Metric value must be a number: myapp.worker.core.offloaded_requests, 

uwsgi --connect-and-read shows data (although lots 0 values)

Suggestions?

Get more metrics, such as harakiri

Hello.

I implemented this plugin to get uwsgi metrics, easy to install and to get metrics, however, it's missing some important metric such as harakiri count.

Is it possible to get more metrics?

thanks

Wrong avg_response_time metric

Hello,

you'd better calculate the average_response_time yourselves, as uWSGI fails to provide a correct implementation of average (and they're not willing to fix it).

See here for the wrong code and here for their position on the matter.