travelping / exometer_influxdb

Exometer reporter for InfluxDB

Exometer InfluxDB reporter

This reporter pushes data to InfluxDB.

Usage

1. Add exometer_influxdb to your list of dependencies in rebar.config:

   {deps, [
       {exometer_influxdb, "0.6.0"}
   ]}.

2. Ensure exometer_influxdb is started before your application:

   {applications, [exometer_influxdb]}.

3. Configure it:

   {exometer_core, [
       {report, [
           {reporters, [
               {exometer_report_influxdb, [{protocol, http},
                                           {host, <<"localhost">>},
                                           {port, 8086},
                                           {db, <<"exometer">>},
                                           {tags, [{region, ru}]}]}
           ]}
       ]}
   ]}.

Available options:

• protocol - http, https or udp for operating with InfluxDB. http by default. If you use udp, check udp_mtu below to avoid {error,emsgsize}.
• host - InfluxDB host. 127.0.0.1 by default.
• port - InfluxDB port. 8086 by default.
• db - Database on InfluxDB for writing data. exometer by default.
• username - Username for authorization on InfluxDB.
• password - Password for authorization on InfluxDB.
• timestamping - Enable timestamping, false by default. To enable timestamping with the reporter you can use true or {true, Precision} where Precision is a unit taken from [n,u,ms,s,m,h]. The default unit is u.
• batch_window_size - Set window size in ms for batch sending. This means the reporter will collect measurements within this interval and send all measurements in one packet. 0 by default.
• udp_mtu - (Used only with protocol == udp.) MTU of the network interface through which UDP packets flow to the host. 65536 by default (Linux loopback interface MTU). Run ifconfig on the machine where this app will run to find it out. Metrics will be sent out if their size (in the Line Protocol format) exceeds this value, even if the current batch_window_size is not over yet. (They will also be sent out at the end of batch_window_size as usual, regardless of their size.)

The following options can be set globally in the reporter config or locally in a specific subscription. The latter case overwrites the first.

• tags - List of tags for each time series. The host is automatically included here.
• series_name - The name of a time series visible within the FROM field. By default this is set to the concatenated elements of the exometer id. Caution: If set in the global reporter config then every time series will have this name.
• formatting - Formatting options to alter the appearance of a series name or tags.

Subscription examples:

{exometer_core, [
    {report, [
        {subscribers, [
            {exometer_report_influxdb, [erlang, memory], total, 5000, [{tags, {tag, value}}]}
         ]}
    ]}
]}.

By default the in InfluxDB visible name of the metric is derived from the exometer id: Here [erlang, memory] is translated to erlang_memory. It is possible to remove an item from this list by naming itself or its position with the from_name keyword. A removed element is then used as tag value:

exometer_report:subscribe(exometer_report_influxdb, [erlang, memory], total, 5000, [{tags, [{tag, {from_name, 2}}]}]).

This will result in a name erlang with the tag pair {tag, memory} (plus the default pair {host, Host}). To disable the removal of elements in the series name you can set:

{formatting, [{purge, [{all_from_name, false}]}]}

Further it might be handy to remove e.g. undefined tag keys or values. This can be done via:

{formatting, [{purge, [{tag_keys, undefined}, {tag_values, undefined}]}]}

Auto subscriptions:

There is capability for making a subscription automatically for each new entry. By default it is off. If you need to enable it in the reporter options and also provide a callback module which handles newly created entries.

{exometer_core, [
    {report, [
        {reporters, [
            {exometer_report_influxdb, [{autosubscribe, true},
                                        {subscriptions_module, exometer_influxdb_subscribe_mod},
                                        {protocol, http},
                                        {host, <<"localhost">>},
                                        {port, 8086},
                                        {db, <<"exometer">>},
                                        {tags, [{region, ru}]}]}
        ]}
    ]}
]}.

The callback module may look like:

-module(exometer_influxdb_subscribe_mod).
-export([subscribe/2]).

subscribe([metric, test], histogram) ->
    Tags = [{tags, [{type, {from_name, 2}}]}],
    [{[metric, test], min, 1000, Tags},
     {[metric, test], max, 1000, Tags},
     {[metric, test], median, 1000, Tags}];
subscribe([metric, test1], histogram) ->
    Tags = [{tags, [{type, {from_name, 2}}]}],
    [{[metric, test1], max, 1000, Tags},
     {[metric, test1], median, 1000, Tags}];
subscribe(_, _) -> [].

subscribe/2 calls for each new entry and it should return a list or just one subscription. Here a single subscription has the following layout:

{exometer_report:metric(), exometer_report:datapoint(), exometer_report:interval(), exometer_report:extra()}

TODO

• Reconfiguration on runtime
• Enhance the formatting options (e.g. concatenation chars, format strings, etc.)

exometer_influxdb's Issues

Metric TimeStamp handling

the timestamp for a given metric should be set at the source. Asyou see with batching, unavailable backends etc etc. the receive timestamp of the backenend, in your case influx, will aways differ from the source timstamp. therefore both are important.

for the sample intervals the same is true. there is a difference how often a application updates the exo metrics push from app->exo. and how often the Reporter sends the stats out to the backend.

If we have the intended HTTP GET "reporter" discussed with @uwiger and @GalaxyGorilla the external GET (pull) interval will dictate the reporting interval. If you have a metric type of exometer_function this will also trigger the collection via the callback as far I understand Exometer correctly

Blank out password when logging connection successful info statement

I don't think the password should be leaked to the logs:

14:56:39.781 [info] InfluxDB reporter connecting success: [{autosubscribe,true},{protocol,https},{host,<<"influxdb.domain.com">>},{username,<<"admin">>},{password,<<"mypasswordishere">>},{port,8086},{db,<<"dev">>},{tags,[]}]

https://github.com/travelping/exometer_influxdb/blob/master/src/exometer_report_influxdb.erl#L108

Remove lager as a dependency

Lager is no longer a dependency of exometer_core. It would be great to remove it from exometer_influxdb as well.

Is there anything internal which is requiring lager as the logger instead something like hut which the core project has moved to?

Optional timestamp

According InfluxDB Line Protocol Syntax Reference timestamps are not required and may be represented on server side.

InfluxDB reporter UDP sending error: {error,emsgsize}

Hi 🙃 Getting this error. The config is:

{exometer, [
    {report, [
        {reporters, [
            {exometer_report_lager, [
                {level, notice},
                {type_map, [{'_',integer}]}
            ]},
            {exometer_report_influxdb, [
                {protocol, udp},
                {host, <<"localhost">>},
                {port, 5268},
                {username, <<"influxdb">>},
                {password, <<"influxdb">>},
                {db, <<"our_app__staging">>},
                {timestamping, {true, ms}},
                {batch_window_size, 100},
                {tags, []}
            ]}
        ]}
    ]}
]},

I'm guessing that the app sometimes generates a burst of metrics, and in 100 ms they are accumulating into a packet that is larger than 65536 bytes (loopback interface MTU).

Could you add | Would you accept a PR with cutting metrics by max UDP packet size?

integer / float type error with histogram

I have created a subscription for influxdb to a histogram. On initialisation the histogram gives back all floats:

{:ok,
 [{:n, 0.0}, {:mean, 0.0}, {:min, 0.0}, {:max, 0.0}, {:median, 0.0}, {50, 0.0},
  {75, 0.0}, {90, 0.0}, {95, 0.0}, {99, 0.0}, {999, 0.0}]}

All values will be reported as float to influxdb. After sending some values some numbers are represented as integers:

{:ok,
 [{:n, 3}, {:mean, 192.397}, {:min, 192.397}, {:max, 192.397}, {:median, 192},
  {50, 192.397}, {75, 192.397}, {90, 192.397}, {95, 192.397}, {99, 192.397},
  {999, 192.397}]}

This results in an error because the reporter detects values when they're being send and there is now a mismatch, see:

14:56:38.771 [warning] InfluxDB reporter got unexpected response with code 400 and body: <<"{\"error\":\"write failed: field type conflict: input field \\\"n\\\" on measurement \\\"ecto_query_exec_time\\\" is type int64, already exists as type float\"}\n">>. Reconnecting ...

I didn't see an option to configure this. I think histograms (because of their initial value) should always be floats. Is this desirable and what should the configuration for it look like? I can invest some time in this and create a PR, but it would be nice to know in which direction it should go.

Can't get exometer_influxdb to work with predefined metrics

A few of us have been getting an error like:

[warning] InfluxDB reporter got trouble when looking [erlang,memory] metric's tag: not_found

You can see the discussion within the comments of this post:
http://milhouseonsoftware.com/2016/05/08/measuring-your-elixir-application/

It seems the metric lookup isn't working. Disabling it as @jbavari has done here seems to be a work around:
jbavari@f14d592

I'm sorry I'm not familiar with Erlang enough to really understand how to fix this.

Long http request to influxdb

Hello. Sorry for my english. I've found really strange bug. I can't imagine how to debug it. I measure schedulers usage each second. Use gauge probe. For example scheduler usage of idle app is 0.1. When I add some load to my application I can see scheduler usage 0.5 with exometer:get_value([metric, scheduler_usage]). But new values from influxdb stay 0.1. Only after few minutes new values reported from exometer_influxdb becomes 0.5. With graphite reporter everything is ok. I can't reproduce it in a simpler example for now.

exometer:new([metric, scheduler_usage], gauge, []),`
spawn_link(fun scheduler_usage/0),

scheduler_usage() ->
    Ts0 = lists:sort(erlang:statistics(scheduler_wall_time)),
    timer:sleep(1000),
    Ts1 = lists:sort(erlang:statistics(scheduler_wall_time)),
    {A, T} = lists:foldl(
        fun({{_, A0, T0}, {_, A1, T1}}, {Ai,Ti}) ->
            {Ai + (A1 - A0), Ti + (T1 - T0)}
        end,
    {0, 0}, lists:zip(Ts0, Ts1)),
    exometer:update([metric, scheduler_usage], A/T),
    scheduler_usage().

  {exometer, [
      {report, [
          {reporters, [
              {exometer_report_influxdb, [
                  {protocol, http}, 
                  {host, <<"some hosy">>},
                  {port, 8086},
                  {db, <<"dev">>}
              ]}
          ]},
          {subscribers, [
              {exometer_report_influxdb, [metric, scheduler_usage], value, 1000, true, []}
          ]}
      ]}
  ]}

lager is missing as a dependency

Lager is missing as a dependency in rebar.config. I wanted to create a PR directly but I'm not sure how you would like to resolve this (since I also see a 'use_hut' branch). Are you okay with fetching lager from hex.pm? Or do you want to pull it from github?

I'm using mix which currently can't resolve the dependencies correctly which causes compilation to fail.

Ps. Are you interested in publishing this package on hex? I would gladly create a PR.

dependency mismatch - rebar3 & hex?

We have a dependency mismatch which could be solved by using hackney from hex.pm in this project. It would require moving to rebar3 as far as I understand. Would you be open to using rebar3 and pointing to hex.pm instead of repo on github on branch master? I can create a PR in that case.

We used override: true in our config but would like to remove that again.

Dependencies have diverged:
* hackney (Hex package)
  the dependency hackney in deps/httpoison/mix.exs is overriding a child dependency:

  > In deps/httpoison/mix.exs:
    {:hackney, "~> 1.8.0", [env: :prod, hex: "hackney", repo: "hexpm", optional: false, manager: :rebar3]}

  > In deps/exometer_influxdb/rebar.config:
    {:hackney, ~r/.*/, [env: :prod, git: "git://github.com/benoitc/hackney.git", branch: "master", manager: :rebar]}

  Ensure they match or specify one of the above in your deps and set "override: true"
** (Mix) Can't continue due to errors on dependencies

Metrics are not sent when protocol is `https`

The InfluxDB reporter supports connecting via https, but the send function only supports http, so metrics are not sent to InfluxDB when the protocol is https.

Memory leak issue

Hi, I have been using exometer_infludb along with my Phoenix app. But I have found that the exometer_report_influxdb plugin doesn't release memory. Currently, its consuming more than300MB of memory and has around 400K messages in its queue. My pool_interval is 3 seconds. Any pointers on why is it exhibiting this behavour?

Tags from metrics name

Exometer doesn't support tags in metric name.

Lets we have metics:

[eradius, '127.0.0.1:8111', requests]
[eradius, '127.0.0.1:9222', requests]

And we want to see it in InfluxDB like this:

eradius_requests, nas=127.0.0.1:8111
eradius_requests, nas=127.0.0.1:9222

I suggest describing tags via Extra parameter when we create a subscription:

exometer_report:subscribe(Reporter, [eradius, '127.0.0.1:8111', requests], value, 500, [{nas, {from_name, 2}}], true).

It means "create a tag called nas from the second element of name and remove it from the name"

This solution works well in exometer infrastructure and has backward compatibility.