phobos182 / collectd-elasticsearch Goto Github PK

Hi,

The collectd-elasticserach plugin seems to be working correctly as i get the below logs:
[2015-12-04 04:56:31] elasticsearch plugin [verbose]: Configured with version=1.0, host=localhost, port=9200, url=http://localhost:9200/_nodes/_local/stats/transport,http,process,jvm,indices,thread_pool
[2015-12-04 04:56:31] elasticsearch plugin [verbose]: Sending value[gauge]: thread_pool.bulk.threads=0
....

But for some reason collectd is 'not' sending the ElasticSearch metrics data to my graphite database. Is there some integration issue with write_http set with json format.

Collectd is able to send other metrics data from genericJMX etc to my graphite database.

My version of collectd is 5.4 on ubuntu 14.04

Please help.
Thanks
Jaskaran

Hi,

I have just run into a very hard to debug problem. After adding elasticsearch.py to my config, I had the error

python plugin: Found a configuration for the "elasticsearch" plugin, but the plugin isn't loaded or didn't register a configuration callback.

It was because I had already installed the ElasticSearch python client globally and so collectd was loading the wrong module. Sure, this is a newbie mistake and I should have used a virtualenv, but I guess maybe renaming elasticsearch.py to elasticsearch_collectd.py could perhaps save some users some headaches ;-)

Thanks !

When using elasticsearch version 0.19.9, the default elasticsearch stats URL that the plugin queries doesn't provide stats for http, process, jvm, and transport, so the plugin fails with the following error:

Sep  3 16:37:14 localhost collectd[16116]: Unhandled python exception in read callback: TypeError: int() argument must be a string or a number, not 'NoneType'
Sep  3 16:37:14 localhost collectd[16116]: read-function of plugin `python.elasticsearch' failed. Will suspend it for 40 seconds.

I've already fixed this in my fork, so I'll send a pull request.

Is this project still being maintained?

It would be extremely helpful to have health stats.
My python-fu being very weak, it might take a while before I do this

$ curl 0:9200/_cluster/health\?pretty
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 11,
  "number_of_data_nodes" : 9,
  "active_primary_shards" : 3148,
  "active_shards" : 8000,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "number_of_pending_tasks" : 0
}

i'm using collectd version 5.4.0, and es version 0.9.x. When loading the plugin the syslog reports it's able to fire the read callback and find all the stats (per this https://gist.github.com/petecheslock/795b708d1fbddc459dc5) But as you can see in gives an error when it tries to write: Filter subsystem: Built-in target write': Dispatching value to all write plugins failed with status -1`

If I uncomment the rrdtool plugin and comment out

<target "write">
        Plugin write_http
</target>

Then the metrics get tracked but into rrd (and not up to librato obviously).
My (sanitized) collectd.conf is in the following gist.

https://gist.github.com/petecheslock/d6c47edd2e32bf35ffb9

So i'm guessing there is something about the write_http plugin that the output of this script doesnt likes, although i'm not sure where to look. Any ideas?

Not everyone uses the default elasticsearch cluster name of 'elasticsearch', so the cluster name should be a configurable option.

I already have a fix for this in my fork, so I'll submit a pull request.

Hi @phobos182 ,

Please may I know which version of collectd you used .
Getting this below error

Traceback (most recent call last):
File "elasti_metric.py", line 254, in
collectd.register_config(configure_callback)
AttributeError: 'module' object has no attribute 'register_config'

Hello,

On the ES 1.0 version you're using /_nodes/stats instead of /_cluster/nodes/_local/stats, then parse only one result whatever the node you're graphing.

Unfortunately, it f#@ up all the indices metrics when the ES cluster is split into routing nodes and data nodes as your plugin may try to track indices related metrics from the query node.

/_nodes/stats is equal to /_nodes/all/stats, using /_nodes/_local/stats which fixes the problem on non data node.

So it there a reason why you use /_nodes/all/stats?