fangli / fluent-plugin-influxdb

A buffered output plugin for fluentd and InfluxDB

Fluent::Plugin::InfluxDB, a plugin for Fluentd

fluent-plugin-influxdb is a buffered output plugin for fluentd and influxDB.

If you are using fluentd as a collector and want to organize your time-series data in influxDB, this is your right choice!

Requirements

fluent-plugin-influxdb	fluentd	ruby
>= 2.0.0	>= v1.0.0	>= 2.3
>= 1.0.0	>= v0.14.0	>= 2.1
< 1.0.0	>= v0.12.0	>= 1.9

Installation

$ fluent-gem install fluent-plugin-influxdb --no-document # for fluentd v1.0 or later
$ fluent-gem install fluent-plugin-influxdb -v "~> 1.0" --no-document # for fluentd v1.0  or later and ruby 2.2 or earlier
$ fluent-gem install fluent-plugin-influxdb -v "~> 0.3" --no-document # for fluentd v0.12 or later and ruby 2.0 or earlier

Ruby 2.0 or earlier

influxdb gem requires cause gem in Ruby 2.0 or earlier. If you want to use fluent-plugin-influxdb with Ruby 2.0 or earlier, you should install cause gem before install fluent-plugin-influxdb.

Usage

Just like other regular output plugins, Use type influxdb in your fluentd configuration under match scope:

@type influxdb

---

Options:

host: The IP or domain of influxDB, separate with comma, default to "localhost"

port: The HTTP port of influxDB, default to 8086

dbname: The database name of influxDB, default to "fluentd". you should create the database and grant permissions first

measurement: The measurement/serise for record insertion. The default is nil.

user: The DB user of influxDB, should be created manually, default to "root"

password: The password of the user, default to "root"

retry: The finite number of retry times. default is infinite

use_ssl: Use SSL when connecting to influxDB. default to false

verify_ssl: Enable/Disable SSL Certs verification when connecting to influxDB via SSL. default to true

time_key: Use value of this tag if it exists in event instead of event timestamp

time_precision: The time precision of timestamp. default to "s". should specify either hour (h), minutes (m), second (s), millisecond (ms), microsecond (u), or nanosecond (ns)

auto_tags: Enable/Disable auto-tagging behaviour which makes strings tags.

tag_keys: The names of the keys to use as influxDB tags.

sequence_tag: The name of the tag whose value is incremented for the consecutive simultaneous events and reset to zero for a new event with the different timestamp

default_retention_policy: The retention policy applied by default. influxdb >= 0.2.3 is required to use this functionality.

retention_policy_key: The name of the key in the record whose value specifies the retention policy. The default retention policy will be applied if no such key exists. influxdb >= 0.2.3 is required to use this functionality.

cast_number_to_float: Enable/Disable casting number to float. influxdb can't mix integer/float value in one measurement. If your pipeline can't unify record value, this parameter may help. Avoid 'field type conflict' error.

Fluentd Tag and InfluxDB Series

influxdb plugin uses Fluentd event tag for InfluxDB series. So if you have events with app.event, influxdb plugin inserts events into app.event series in InfluxDB.

If you set measurement parameter, use its value instead of event tag.

Configuration Example

<match mylog.*>
  @type influxdb
  host  localhost
  port  8086
  dbname test
  user  testuser
  password  mypwd
  use_ssl false
  time_precision s
  tag_keys ["key1", "key2"]
  sequence_tag _seq
</match>

Cache and multiprocess

fluentd-plugin-influxdb is a buffered output plugin. So additional buffer configuration would be (with default values):

<buffer>
  @type memory
  chunk_limit_size 524288 # 512 * 1024
  chunk_limit_records 1024
  flush_interval 60
  retry_limit 17
  retry_wait 1.0
  num_threads 1
</buffer>

The details of BufferedOutput is here.

---

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Licence

This package was distributed under MIT licence, see LICENCE file for details.

This plugin was written by Fang.Li and was inspired by Uken's elasticsearch plugin.

fluent-plugin-influxdb's Issues

Plugin prevents fluentd from shutting down when database is not online.

I had to make the following change, otherwise the thread will forever attempt to connect to the database. when it is not online.

def shutdown
    @influxdb.stop!
end

Setting 'tag_keys' dynamically

In the work that I am doing, I do not know in advance the tags to be used in tag_keys. For every packet that is received I can look at the data and decide what the tags could be. Is there any way that I can set the 'tag_keys' per record that I am will be sending to influxDB?

For example, I could get a packet which could transform to {'interface-name': 'abc', 'interface-bandwidth': 1234, }. For this the tag should be 'interface-name'. Next I could get a packet which could transform to {'arp-ip': '1.2.3.4' and 'arp-timeout': 34}. Here the index should be 'arp-ip'. All of the packets are coming at the same time, so the timestamp would be the same and these should of to the same measurement. Above records are just example to simplify the use case.

About default_retention_policy and retention_policy_key

I don't know hot to write a "defalut_retention_policy" or "About default_retention_policy and ".
Can U give an example,
such as keep date for 30 daya , how can I write for "defalut_retention_policy"

Plugin unable parse message and suppressed

Dear fangli,

Today i view td-agent log and see error log: unable to parse... . I sure my format of message's correct, But plugin say unable to parse ( miss "," between "ran=408880 ssec=1448935558" ). And buffer stuck this error message cannot retry (because of error) or by pass error message when i restart td-agent. I must delete buffer. Please check & fix it

P/s: I ran td-agent with influxdb-plugin in 3 days but today it error. I delete buffer & restart , td-agent work back

2015-12-01 10:13:03 +0700 [warn]: temporarily failed to flush the buffer. next_retry=2015-12-01 10:34:00 +0700 error_class="InfluxDB::Error" error="unable to parse 'cdn.stream1.mecloud.vn,node=mc11,schema=http,domain=cdn.stream1.mecloud.vn,uri=/clip/2015/12/1/8/19/77050ecf7bccc5338afb810797050cd6_360.mp4,status=206,cache=HIT,ip=113.162.174.190,nation=VN,region=Ho\ Chi\ Minh,agent=GET\ /clip/2015/12/1/8/19/77050ecf7bccc5338afb810797050cd6_360.mp4?e=1448949944&s=8y0vl3kSYUgw7jkY-sXqAw\ HTTP/1.1,ran=408880 ssec=1448935558,byte=1188298,ust=0,usl=0,hit=1,xx2=1,xx3=0,xx4=0,xx5=0,byteos=0,bytevn=1188298,req=1,reqvn=1,reqos=0 1448935558': missing tag value\n" plugin_id="object:3feac88730c8"
2015-12-01 10:13:03 +0700 [warn]: suppressed same stacktrace

Disordered writes in a connection-faiilure environment

Hi!

I'm trying to insert in some sequentials writes in InfluxDB comming from a "not stable connection environment". Td-agent, basically reads from rabbitmq, and insert into a measurement schema.
The configuration in the server side is this:

<source>
  @type amqp
  host localhost
  port 5672
  vhost /
  user admin
  pass mypass
  queue health_info
  format json
  durable true
  tag_key true
</source>

<match **>
  @type influxdb
  dbname health
  user admin
  password mypass
  use_ssl false
  retry 0
  num_threads 1
  sequence_tag _seq
</match>

The problem comes when i cut off the connection between my client and server: when RabbitMQ is reconnected, all the stored messages pass between host and server in order, and they are diferents (they have an additional "timestamp" field) but they are written totally disordered at InfluxDB

Some example messages:

{"disk_usage": 2, "cpu_usage": 1.0, "percent_mem": 20.9, "disk_percent": 57.7, "used_mem": 193, "timestamp": "2017-01-07T17:41:32Z"}
{"disk_usage": 2, "cpu_usage": 0.5, "percent_mem": 20.9, "disk_percent": 57.7, "used_mem": 193, "timestamp": "2017-01-07T17:41:42Z"}

Some considerations:
a) This is not happening when connection is not cutted off between AMQP server altough you're writing these at InfluxDB. All messages are written in order.
b) RabbitMQ is passing once message (line) at time.
c) If i replace InfluxDB for file output plugin, this works correctly and all lines are written sequentially onto the file, as we were passed across.

Can this be a bug?
Thanks!

Does plugin use line protocol for json format source?

I am trying to use the fluentd docker log driver. I write stdout as json and have fluentd forward that to influxdb using this plugin.

fluentd.conf

<source>
  @type forward
  format json
  port 24224
</source>

<match **>
  @type influxdb
  host  influxdb
  port  8086
  dbname test
  user  admin
  password  admin
  use_ssl false
  measurement pingdom
  flush_interval 1s
</match>

It seems as though it just forwards the json that it gets from docker. Most like because of the source setup, but since current influxdb only supports line protocol, how can we we forward from a source of json and have fluentd format that to line protocol?

auto_tag functionality

hi,
i'm trying to ingest telemetry data from my juniper router into my influx db. i have a few fieldKeys that are of type string. i want to convert these fieldKeys to influx tags by using the auto_tag option, but this is not working.

if i use the tag_keys option then i have the same key saved as a field in influx and as a tag.

example this is what the field keys looks like in my influx db when i ingest the data:

> show field keys from interfaces
name: interfaces
fieldKey         fieldType
--------         ---------
device           string
egress_queue     integer
interface        string
interface_parent string
type             string
value            integer

if i use the tag_key option in my fluent.conf file as listed below, i then end up with a key being a fieldKey and tagKey in my influx db. this gives me problem when i use grafana. i hope i stated that clearly.

i have the following fluent.conf file configured:

#############
## INPUT ##
#############

<source>
   @type udp
   tag jnpr.jvision
   #tag interfaces
   format juniper_jti
   output_format structured
   port 50051
   bind 0.0.0.0
</source>


##############
## OUTPUT ##
##############

<match jnpr.**>
#<match interfaces>
   @type copy

   <store>
      type influxdb
      host localhost
      port 8086
      dbname testfluent
      measurement interfaces
      #user juniper
      #password juniper
      #value_keys ["value"]
      #field_keys ["value"]
      #tag_keys ["interface_parent","device","type"]
      auto_tags enable
      buffer_type memory
      # buffer_chunk_limit 524288 # 512 * 1024
      # buffer_queue_limit 1024
      flush_interval 2
      # retry_limit 17
      # retry_wait 1.0
      # num_threads 1
   </store>
</match>

## match tag=debug.** and dump to console
<match jnpr.**>
#<match interfaces>
  @type stdout
  @id stdout_output
</match>

is there a way to convert a fieldKey to a tagKey and have influx db store it as such without the same key being a field and a tag in influx? thanks!

Adding support for InfluxDB 2?

I wanted to reach out and figure out if you would accept a PR to support the latest InfluxDB 2.0 apis?

We have a plugin ready to go, I think it would just need to be pr'd and reviewed here: https://github.com/influxdata/influxdb-plugin-fluent/blob/master/lib/fluent/plugin/out_influxdb2.rb

I'd love to figure out how to combine these two plugins.

Backslashes are not properly escaped hence result in "unable to parse ... bad timestamp" error on InfluxDB

Original log line like this;

Parameters: {"commit_message"=>"Revert \"Merge Branch 'hot_fix' into 'master'\"\r\n\r\nSee MR !2"}

should be encoded on the line protocol like this;

production.log,message="Parameters ...snip... into 'master'\\\"\\r\\n\\r\\nSee MR !2\"}"

but out_influxdb encodes like this;

production.log,message="Parameters ...snip... into 'master'\\"\r\n\r\nSee MR !2\"}"

which InfluxDB can not parse and returns "unable to parse ... bad timestamp" error.

For your reference, out_file encodes it just right;

production.log {"message":"Parameters ...snip... into 'master'\\\"\\r\\n\\r\\nSee MR !2\"}"}

Versions:
fluent-plugin-influxdb:1.0.0.rc2
fluentd 0.14
influxdb 1.2

How can i use database come from record ?

i use fluent-plugin-influxdb:1.0.0.rc3

<filter container.*>
@type record_transformer
remove_keys Name
enable_ruby true

time ${Time.parse(record["time"]).to_i}
test_value testdb

<match container.*>
@type influxdb
host influxdb
port 8086
dbname ${test_value}
user writer
password YWM0ZGY2M2Y4NjY0
use_ssl false
auto_tags true
buffer_chunk_limit 8m
buffer_queue_limit 32
buffer_queue_full_action drop_oldest_chunk
flush_interval 1s
num_threads 4

#0 Connecting to database: ${test_value}, host: influxdb, port: 8086, username: writer, use_ssl = false, verify_ssl = true

Number/float field are pushed into influxdb as string

Hi,
I try to use this plugin to convert nignx access log into metrics and push it into influxdb.
The nginx access log format likes the following:

[26/Oct/2019:17:21:40 +0000] 302 "[email protected]" "GET / HTTP/1.0" 4801 "-" "-" 0.631 0.631 .

I expect the status code(302), request time(0.631) and response time(0.631) should be saved into influxdb AS number/float data type. But actually all of them are stored as string.

> select * from fe_portal
name: fe_portal
time                agent        code hostname               method path project   referer region          remote    request_time response_time size tailed_path                     timestamp                  user version
----                -----        ---- --------               ------ ---- -------   ------- ------          ------    ------------ ------------- ---- -----------                     ---------                  ---- -------
1572110977000000000 GoogleHC/1.0 302  wf20-td-agent-test-1-0 GET    /    prod-test -       asia-southeast1 127.0.0.1 0.630        0.630         4801 /var/log/pan/my-access-test.log 26/Oct/2019:17:29:21 +0000 -    HTTP/1.0
> select request_time::float from fe_portal
> select request_time::string from fe_portal
name: fe_portal
time                request_time
----                ------------
1572110977000000000 0.630
>

My fluentd config likes the following:

<source>
  @type tail
  pos_file /var/log/td-agent/nginx-influx.pos
  path_key tailed_path
  @id input_tail
  path /var/log/pan/*access*.log
  tag fe_portal
  <parse>
    @type regexp
    expression /^\[(?<timestamp>[^\]]*)\] (?<code>[^ ]*) "(?<user>[^ ]*)@(?<remote>[^ ]*)" "(?<method>\S+)(?: +(?<path>[^\ "]*?)(?: +\S*)?) (?<version>\S+/\S+)" (?<size>[\d.]+) "(?<referer>[^ ]*)" "(?<agent>[^\"]*)" (?<request_time>[\d.-]+) (?<response_time>[\d.-]+).*$/
    time_format %d/%b/%Y:%H:%M:%S %z
  </parse>
</source>
<filter fe_portal>
  @type record_transformer
  auto_typecast true
  <record>
    hostname ${hostname}
    project prod-test
    region asia-southeast1
  </record>
</filter>
<match fe_portal>
  type influxdb
  num_threads 8
  dbname test
  host influxdb
  port 8086
  flush_interval 10s
  cast_number_to_float true
  @id influxdb
  tag_keys ["method","path", "code", "region", "hostname"]
</match>

one second has only one data

My timestamp is 2015-12-13T07:39:31Z, but when insert to influxdb has only one message in one second. In fact, one second has more than one message.

fluent-plugin-influxdb (0.2.2)
influxdb (0.2.3)
influxdb server : 0.9.6.1

By the way: the config "tag_keys" isn't efficient.

Feature Request: Option to write Boolean as Integer

Cuz now, the influxdb supports boolean is still very bad,
but most metrics from fluent-bit use boolean value.
Still lacks transform func without using kapacitor

Document the format of a message

Im trying to figure out how a line in a file must be formated to be forwarded to influx.

After 3 hours with guessing, looking at tests, previous issues and looking a little bit into the code, i think this ticket is my last hope.

Could you add a line to the config section with a example line? This would be a big help!

I think a good example would be the InfluxDB example:

cpu_load_short,host=server02 value=0.67
cpu_load_short,host=server02,region=us-west value=0.55 1422568543702900257
cpu_load_short,direction=in,host=server01,region=us-west value=2.0 1422568543702900257

What should my config and input line look like, to get the same data like the influxdb example, into my influx?:

<source>
    type tail
    path /tmp/dummy.log
    pos_file /tmp/dummy.log.pos
    tag dummy.test
    format none
  </source>
  <match dummy.*>
    type influxdb
    host 10.10.10.1
    port 8086
    dbname db
    user admin
    password xxxxxx
    use_ssl false
    tag_keys ["host"]
    time_precision s
  </match>

Measurement configruation

Is there a way to choose or override the default measurement/series name? Currently it defaults to the fluentd tag which is not ideal in some configruations.

escape hell - link bug reported on fluentd

Please check fluent/fluentd#1736
( I quite believe the issue with wrong escapes happen in the influx plugin)

Database presence check is not work with non-admin user

I got an error message like following after I upgrade this plugin to v2.5:

2016-04-22 15:25:57 +0900 [error]: unexpected error error_class=InfluxDB::AuthenticationError error=#<InfluxDB::AuthenticationError: {"error":"error authorizing query: (username) not authorized to execute statement 'SHOW DATABASES', requires admin privilege"}>

I confirmed if I add admin privilege to that user, this error is not shown and the data is imported to InfluxDB correctly. But in my opinion, it's too much for data collection process.

Add formatter functionality?

It would be useful if this plugin could act as a Fluent formatter plugin for Influx Line Protocol. It would extend its usefulness to do things like putting ILP onto a Kafka topic.

Undefined method "nsec"

When trying to run the fluent-plugin in 1.0.0-rc1 i get the following error:

Error log:

2017-02-04 21:12:14 +0000 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `nsec' for 1486242734:Fixnum\nDid you mean?  inspect" tag="metric.docker.upbeat_allen"
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-influxdb-1.0.0.rc1/lib/fluent/plugin/out_influxdb.rb:217:in `precision_time'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-influxdb-1.0.0.rc1/lib/fluent/plugin/out_influxdb.rb:108:in `format'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/output.rb:876:in `block in handle_stream_simple'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event.rb:193:in `block in each'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event.rb:192:in `each'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event.rb:192:in `each'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/output.rb:875:in `handle_stream_simple'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/output.rb:770:in `execute_chunking'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/output.rb:707:in `emit_buffered'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event_router.rb:153:in `emit_events'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event_router.rb:90:in `emit_stream'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event_router.rb:81:in `emit'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-grep-0.3.4/lib/fluent/plugin/out_grep.rb:77:in `block (2 levels) in emit'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-grep-0.3.4/lib/fluent/plugin/out_grep.rb:70:in `catch'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-grep-0.3.4/lib/fluent/plugin/out_grep.rb:70:in `block in emit'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event.rb:107:in `each'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-grep-0.3.4/lib/fluent/plugin/out_grep.rb:69:in `emit'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/compat/output.rb:162:in `process'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/output.rb:696:in `emit_sync'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event_router.rb:90:in `emit_stream'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event_router.rb:81:in `emit'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:377:in `on_message'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:231:in `block in handle_connection'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:272:in `block (3 levels) in read_messages'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:271:in `feed_each'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:271:in `block (2 levels) in read_messages'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:280:in `block in read_messages'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:560:in `on_read'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/cool.io-1.4.6/lib/cool.io/io.rb:123:in `on_readable'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/cool.io-1.4.6/lib/cool.io/io.rb:186:in `on_readable'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run_once'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run'
  2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:190:in `run'
2017-02-04 21:12:14 +0000 [warn]: out_grep: NoMethodError undefined method `nsec' for 1486242734:Fixnum
Did you mean?  inspect /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-influxdb-1.0.0.rc1/lib/fluent/plugin/out_influxdb.rb:217:in `precision_time'

Fluentd Dockerfile

FROM centos:7

MAINTAINER Jimmi Dyson <[email protected]>

ENTRYPOINT ["/start-fluentd"]

ENV LD_LIBRARY_PATH /opt/rh/rh-ruby23/root/usr/lib64
ENV FLUENTD_VERSION 0.14.8
ENV ELASTICSEARCH_HOST es-logging.default.svc

RUN yum update -y && \
    yum install -y centos-release-scl-rh && \
    yum install -y scl-utils make gcc bzip2 rh-ruby23 rh-ruby23-ruby-devel && \
    yum clean all


RUN scl enable rh-ruby23 'gem update --system --no-document' && \
    scl enable rh-ruby23 'gem install --no-document json_pure jemalloc' && \
    scl enable rh-ruby23 "gem install --no-document fluentd -v ${FLUENTD_VERSION}" && \
    ln -s /opt/rh/rh-ruby23/root/usr/local/bin/* /usr/bin


ENV ELASTICSEARCH_HOST es-logging.default.svc
RUN touch /var/lib/rpm/* && yum install -y gcc-c++ && yum clean all

RUN scl enable rh-ruby23 'gem install --no-document fluent-plugin-elasticsearch -v 1.9.2' && \
    scl enable rh-ruby23 'gem install fluent-plugin-influxdb -v 1.0.0.rc1 --no-document' && \
    scl enable rh-ruby23 'gem install fluent-plugin-parser' && \
    scl enable rh-ruby23 'gem install fluent-plugin-grep' && \
    scl enable rh-ruby23 'gem cleanup fluentd'

ADD start-fluentd /start-fluentd

Fluentd config:

<source>
  type forward
  port 24224
  bind 0.0.0.0
</source>

<match docker.**>
  @type grep
  regexp1 log resource_request
  add_tag_prefix metric
</match>

<match metric.docker.**>
  @type influxdb
  measurement measurement
  host  10.93.1.21
  port  8086
  dbname db1
  user  root
  password  somepassword
  use_ssl false
  time_precision s
  tag_keys ["key1", "key2"]
  sequence_tag _seq
</match>

<match metric.docker.**>
  @type stdout
</match># input plugin that exports metrics

Stdout:

{"measurement":"resource_request","tags":{"name":"asd","value":"asd","key1":"test"},"values":{"response_time":1,"status":200}}

input number data but output string

I don't know if I do wrong something.
I want to put numeric data on influxdb with fluentd, but all data will save as a string type.

My fluentd conf file is like this:
source:

<source>
    @type http
    port 12102
    format tsv
    keys string1.string2,number1,number2
    delimiter |
</source>

Match:

<match test>
                @type influxdb
        dbname test
        flush_interval 10s
        host localhost
        port 8086
</match>

Allow empty strings for fields

c281115 ensures that no fields are empty strings. Although an empty string is an invalid value for a tag, InfluxDB does allow them for string fields.

For example, this is perfectly valid: test,foo=bar fizz="" 1465839830100400200

backslash escaping issue

Hi,

I found that the backslash escaping in the tag/field values was not working correctly as expected.

With config:

<match mytag>
  @type copy
  <store>
    @type stdout
  </store>
  <store>
    @type influxdb
    host influxdb
    port 8086
    dbname mydb
    tag_keys ["hostname", "v.device_name", "v.mount_point"]
  </store>
</match>

And input record (from metricbeat):

{"hostname":"my-machine","v.device_name":"D:\\","v.mount_point":"D:\\","v.total":26840395776,"v.free":5271150592,"v.available":5271150592,"v.files":0,"v.free_files":0,"v.used.pct":0.8036,"v.used.bytes":21569245184}

Will cause the following exception:

#0 taking back chunk for errors. chunk="55a01ef2e4d034e413d394ffd26be22c"
#0 failed to flush the buffer. retry_time=0 next_retry_seconds=2017-09-25 11:34:03 +0000 chunk="55a01ef2e4d034e413d394ffd26be22c" error_class=InfluxDB::Error error="{\"error\":\"unable to parse 'system.filesystem,hostname=my-machine,v.device_name=D:\\\\,v.mount_point=D:\\\\ v.total=26840395776i,v.free=5271150592i,v.available=5271150592i,v.files=0i,v.free_files=0i,v.used.pct=0.8036,v.used.bytes=21569245184i 1501729298688': invalid tag format\"}\n"
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/client/http.rb:85:in `resolve_error'
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/client/http.rb:35:in `block in post'
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/client/http.rb:53:in `connect_with_retry'
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/client/http.rb:26:in `post'
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/query/core.rb:78:in `write'
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/query/core.rb:57:in `write_points'
#0 /fluentd/plugins/out_influxdb2.rb:189:in `write'
#0 /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/plugin/output.rb:1061:in `try_flush'
#0 /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/plugin/output.rb:1286:in `flush_thread_run'
#0 /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/plugin/output.rb:438:in `block (2 levels) in start'
#0 /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

The exception message shows that the record has been wrongly escaped:
(double escaped backslashes and a comma , before v.total disappeared)

v.device_name=D:\\\\,v.mount_point=D:\\\\ v.total=26840395776i,v.free=5271150592i,v.available=5271150592i

I also checked the plugin code and printed the hash point before calling @influxdb.write_points():

{:timestamp=>1501729298688, :series=>"system.filesystem", :values=>{"v.total"=>26840395776, "v.free"=>5271150592, "v.available"=>5271150592, "v.files"=>0, "v.free_files"=>0, "v.used.pct"=>0.8036, "v.used.bytes"=>21569245184}, :tags=>{"hostname"=>"my-machine", "v.device_name"=>"D:\\", "v.mount_point"=>"D:\\"}}

I guess that it is caused by influxdb-ruby, but I think that asking here is a good start.
Thanks.

Environment:

fluentd 0.14.21
fluent-plugin-influxdb 1.0.0.rc3
influxdb-ruby 0.3.16

New version of influxdb fails to import `cause` gem

My temporary workaround is to

gem install influxdb --version 0.2.3

before

fluent-gem install fluent-plugin-influxdb

That may not be the exact version where the issue was introduced, but it works.

Here's a log from a start->finish run with the failure:

2016-05-18 18:59:56 +0000 [info]: reading config file path="/etc/fluentd/fluentd.conf"
2016-05-18 18:59:56 +0000 [info]: starting fluentd-0.12.23
2016-05-18 18:59:56 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.5.0'
2016-05-18 18:59:56 +0000 [info]: gem 'fluent-plugin-exclude-filter' version '0.0.1'
2016-05-18 18:59:56 +0000 [info]: gem 'fluent-plugin-flatten-hash' version '0.2.0'
2016-05-18 18:59:56 +0000 [info]: gem 'fluent-plugin-grep' version '0.3.4'
2016-05-18 18:59:56 +0000 [info]: gem 'fluent-plugin-influxdb' version '0.2.7'
2016-05-18 18:59:56 +0000 [info]: gem 'fluentd' version '0.12.23'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- cause (LoadError)
    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /var/lib/gems/1.9.1/gems/influxdb-0.3.0/lib/influxdb/client.rb:2:in `<top (required)>'
    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /var/lib/gems/1.9.1/gems/influxdb-0.3.0/lib/influxdb.rb:19:in `<top (required)>'
    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:60:in `require'
    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:60:in `rescue in require'
    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:35:in `require'
    from /var/lib/gems/1.9.1/gems/fluent-plugin-influxdb-0.2.7/lib/fluent/plugin/out_influxdb.rb:3:in `<top (required)>'
    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin.rb:172:in `block in try_load_plugin'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin.rb:170:in `each'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin.rb:170:in `try_load_plugin'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin.rb:130:in `new_impl'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin.rb:59:in `new_output'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin/out_copy.rb:46:in `block in configure'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin/out_copy.rb:37:in `each'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin/out_copy.rb:37:in `configure'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/agent.rb:132:in `add_match'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/agent.rb:63:in `block in configure'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `each'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `configure'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:86:in `configure'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/engine.rb:129:in `configure'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/engine.rb:103:in `run_configure'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:483:in `run_configure'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:154:in `block in start'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `call'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `main_process'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:333:in `block in supervise'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `fork'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `supervise'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:150:in `start'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/bin/fluentd:5:in `<top (required)>'
    from /usr/local/bin/fluentd:23:in `load'
    from /usr/local/bin/fluentd:23:in `<main>'
2016-05-18 18:59:56 +0000 [info]: process finished code=256
2016-05-18 18:59:56 +0000 [warn]: process died within 1 second. exit.

sanity check for parameters

It would be great to have sanity check for parameters passed to plugin.

• empty/invalid host/port/db ...

At the moment when host is empty, fluentd loads the config and silently hangs.

gemspec for 0.1.8 causes issues

The gemspec for 0.1.8 specifies the dependency of influxdb as ~>0.1.8 which causes it to install 0.2.0 which is not compatible with Influx 0.8. I believe it should be changed to either just 0.1.8 or ~<0.1.8.

in rc2 version error: fluent-logger for nodejs (set milliseconds:1):report error:error="undefined method `nsec' for 1488276994397.0:Float\nDid you mean? inspect

I use fluent-logger nodejs.
var config = {
host: 'localhost',
port: 24224,
timeout: 3.0,
milliseconds: 1
};

when I write log to influxdb through fluent(support milliseconds? support nodejs?)

error came:

2017-02-28 10:16:37 +0000 [error]: #0 unexpected error on reading data host="172.17.0.1" port=33606 error_class=NoMethodError error="undefined method `nsec' for 1488276994397.0:Float\nDid you mean? inspect"

2017-02-28 10:16:37 +0000 [error]: #0 suppressed same stacktrace
49131-08-17 16:48:06.000000000 +0000 test: {"level":"info","message":"newdada:"}

2017-02-28 10:18:49 +0000 [warn]: #0 emit transaction failed: error_class=NoMethodError error="undefined method `nsec' for 1488277126086.0:Float\nDid you mean? inspect" tag="test"

Gracefully Upgrading to InfluxDB 0.9 Support

The influxdb-ruby library is moving to 0.9 support. Influxdb-ruby v0.1.9 will be the last version that supports 0.8. Without a version constraint on fluent-plugin-influxdb for influxdb-ruby, many new production deployments that have not upgraded to 0.9 will fail un-expectantly.

I recommend that we add a constraint to use only influxdb-ruby v0.1.9 so that we will not fail to work with 0.8 when the next version is released. It might be useful to start versioning fluent-plugin-influxdb such that version 0.1.* works with influxdb 0.8 while a fluent-plugin-influxdb 0.2.* version is compatible with influxdb 0.9. If people use Semver correctly, this should help so that we don't get any surprises in production.

Plugin should refuse to send empty tag values to InfluxDB Ruby client

Considering that sending an InfluxDB line with empty tag values such as (notice the empty ident tag):

syslog,host=fwtpcore1b,ident= message=\"BLAHBLAHBLAH\" 1449233580

will result in InfluxDB returning a missing tag value error, I would say that the InfluxDB Ruby client should omit tags with empty values in the line altogether. InfluxCommunity/influxdb-ruby#124 has been opened for this purpose.

That being said, ideally the Fluentd InfluxDB plugin should also prevent sending any empty tag to the Ruby client in the first place in order to catch the error as early as possible.

I have come upon this issue in cases where Syslog message were received with no ident field while having Fluentd output to InfluxDB.

memory leak if influxdb goes down

We are running influxdb plugin with config:
<match metrics.*>
@type influxdb
host 10.47.246.7
port 8086
dbname test
use_ssl false
time_key time
auto_tags
time_precision ms
retry 3

@type memory
# chunk_limit_size 524288 # 512 * 1024
# chunk_limit_records 1024
flush_interval 10
retry_limit 3
retry_wait 1.0
num_threads 1


json

when we stop influxdb we see that fluentd is leaking memory:

any ideas?

problem on regexp, influxdb0.10

Hi all i'm receiving a lot of error for 2 type of tail:

The first:

2016-03-02 19:32:53 +0100 [warn]: temporarily failed to flush the buffer. next_retry=2016-03-02 19:32:54 +0100 error_class="InfluxDB::Error" error="{\"error\":\"partial write:\\nunable to parse 'nginx.access.log,daemon=nginx:,server=10.0.1.164,host=188.95.76.247,date=02/Mar/2016:18:32:40\\\\ +0000,httprequest=GET,request=/voucher/view/123621?pdf=1,protocol=HTTP/1.1,postrequest=-,statuscode=200,pagesize=49480,referrer=-,useragent=Amazon\\\\ CloudFront,renderizepage=0.814\\\\ 0.814\\\\ . htacces1=\\\"-\\\",htacces2=\\\"-\\\" 1456943562': invalid tag format\\nunable to parse 'nginx.access.log,daemon=nginx:,server=10.0.4.140,host=66.249.66.105,date=02/Mar/2016:18:32:41\\\\ +0000,httprequest=GET,request=/it/roma/tour-di-castelli-e-palazzi-c/?_escaped_fragment_=,protocol=HTTP/1.1,postrequest=-,statuscode=200,pagesize=16352,referrer=-,useragent=Amazon\\\\ CloudFront,renderizepage=0.660\\\\ 0.626\\\\ . htacces1=\\\"-\\\",htacces2=\\\"-\\\" 1456943563': invalid tag format\\nunable to parse 'nginx.access.log,daemon=nginx:,server=10.0.4.12,host=79.109.5.14,date=02/Mar/2016:18:32:51\\\\ +0000,httprequest=GET,request=/frontendV3/PROD/fonts/glyphicons-halflings-regular.woff,protocol=HTTP/1.1,postrequest=-,statuscode=200,pagesize=23292,referrer=https://www.musement.com/es/granada/alhambra-v/?gclid=CNC3yrzRossCFfMV0wod9XUIWw,useragent=Amazon\\\\ CloudFront,renderizepage=0.000\\\\ -\\\\ . htacces1=\\\"-\\\",htacces2=\\\"-\\\" 1456943573': invalid tag format\"}\n" plugin_id="object:3fbe5c5716c4"

the second:

2016-03-02 19:32:32 +0100 [warn]: failed to flush the buffer. error_class="InfluxDB::Error" error="{\"error\":\"unable to parse 'track.access.log,host=47.60.45.56,date=2016-03-02\\\\ 19:32:24\\\\ +0100\\\\ ,httprequest=GET,request=track.gif,env=prod\\u0026,url=https%253A%252F%252Fm.musement.com%252Fes%252Fgranada%252Falhambra-entradas-sin-colas-y-visita-guiada-por-la-manana-4072%252Fbooking,pagetype=frontend_mobile_event_booking,referrer=https%253A%252F%252Fm.musement.com%252Fes%252Fgranada%252Falhambra-entradas-sin-colas-y-visita-guiada-por-la-manana-4072%252F,locale=es,hitid=cb0bd867d219e6abd0796f0a6ba69e49-1456943543225,maid=,currency=EUR,session=ad00bc3mlg7a0vvqbh1jcp4tk1,countryId=161,cityId=174,venueId=,eventsId=4072,cart=,transactionId=,transactionTotal=,transactionMrgn=,transactionProducts=,customer=,eventName=event,eventValue=calendar-date,eventAttribute=open,httpprotocol=HTTP/1.1,httpcode=200,size=1443,cityname=Valencia,regionname=Comunidad\\\\ Valenciana,countryname=ES,referer=https://m.musement.com/es/granada/alhambra-entradas-sin-colas-y-visita-guiada-por-la-manana-4072/booking\\\\\\\",useragent=Mozilla/5.0\\\\ (Linux;\\\\ Android\\\\ 5.1.1;\\\\ ALE-L21\\\\ Build/HuaweiALE-L21)\\\\ AppleWebKit/537.36\\\\ (KHTML\\\\,\\\\ like\\\\ Gecko)\\\\ Chrome/48.0.2564.95\\\\ Mobile\\\\ Safari/537.36\\\\\\\" webid=\\\"cb0bd867d219e6abd0796f0a6ba69e49\\\",useragentFALSE=\\\"Mozilla/5.0%20(Linux;%20Android%205.1.1;%20ALE-L21%20Build/HuaweiALE-L21)%20AppleWebKit/537.36%20(KHTML\\\\,%20like%20Gecko)%20Chrome/48.0.2564.95%20Mobile%20Safari/537.36\\\" 1456943544': missing tag value\\nunable to parse 'track.access.log,host=37.14.34.97,date=2016-03-02\\\\ 19:32:25\\\\ +0100\\\\ ,httprequest=GET,request=track.gif,env=prod\\u0026,url=https%253A%252F%252Fwww.musement.com%252Fes%252Froma%252Fmuseos-vaticanos-v%252F%253Fgclid%253DCj0KEQiAu9q2BRDq3MDbvOL1yaYBEiQAD6qoBsu2fRcJDEwRwsNK7qQtYd6tJXSn1PxHDrKayAxLIEgaAh6z8P8HAQ%2526gclsrc%253Daw.ds,pagetype=frontend_venue,referrer=https%253A%252F%252Fwww.google.es%252F,locale=es,hitid=f154c32f575f635202e1d53351eb4fa1-1456943495070,maid=,currency=EUR,session=,countryId=82,cityId=2,venueId=164,eventsId=,cart=,transactionId=,transactionTotal=,transactionMrgn=,transactionProducts=,customer=,eventName=venue,eventValue=calendar-date,eventAttribute=open,httpprotocol=HTTP/1.1,httpcode=200,size=6474,cityname=Mislata,regionname=Comunidad\\\\ Valenciana,countryname=ES, [...] 

this is my config for the first error

<source>
type tail
  path /log/admin/access.log
  pos_file /log/td-agent/admin.access.log.pos
  tag nginx.access.log
  format /^(?<daemon>[^ ]*)\s*(?<server>[^ ]*)\s*(?<host>[^ ]*)\s*(?<htacces1>[^ ]*)\s*(?<htacces2>[^ ]*)\s*\[(?<date>[^ ]*\s\D[0-9]*)\]\s*\"(?<httprequest>[^ ]*)\s*(?<request>[^ ]*)\s*(?<protocol>[^ ]*)\"\s*\"(?<postrequest>.*?)\"\s*(?<statuscode>[^ ]*)\s*(?<pagesize>[^ ]*)\s*\"(?<referrer>[^ ]*)\"\s*\"(?<useragent>.*?)"\s*(?<renderizepage>.*)$/
  time_format %d/%b/%Y:%H:%M:%S %z
</source>
<match nginx.access.log.**>
  type influxdb
  host  pippo
  port  8086
  dbname access_log
  user  pippo
  password  pippo
  use_ssl false
  time_precision s
  tag_keys ["daemon", "server", "host", "htaccess1", "htaccess2", "date", "httprequest", "request", "protocol", "postrequest", "statuscode", "pagesize","referrer", "useragent", "renderizepage"]
  sequence_tag _seq
  flush_interval 10
  retry_limit 3
</match>

and this for the second:

<source>
    type tail
    path /log/track/access.log
    pos_file /log/td-agent/grafanatrack.access_log.pos
    tag track.access.log
    format /^(?<host>[^ ]*)[\s^-]*\s(?<date>[^ ]*\s[0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}\s*[\W0-9]{1,5}[^"]*)[\s^\W]*(?<httprequest>[^ ]*)\s*\W(?<request>[a-z]+\.[a-z]+)\?*env=(?<env>.*?)&;&url=(?<url>.*?)&;&page_type=(?<pagetype>.*?)&;&referrer=(?<referrer>.*?)&;&locale=(?<locale>.*?)&;&id=(?<webid>.*?)&;&hitid=(?<hitid>.*?)&;&maid=(?<maid>.*?)&;&agent=(?<useragentFALSE>.*?)&;&currency=(?<currency>.*?)&;&session=(?<session>.*?)&;&countryId=(?<countryId>.*?)&;&cityId=(?<cityId>.*?)&;&venueId=(?<venueId>.*?)&;&eventsId=(?<eventsId>.*?)&;&cart=(?<cart>.*?)&;&transactionId=(?<transactionId>.*?)&;&transactionTotal=(?<transactionTotal>.*?)&;&transactionMrgn=(?<transactionMrgn>.*?)&;&transactionProducts=(?<transactionProducts>.*?)&;&customer=(?<customer>.*?)&;&eventName=(?<eventName>.*?)&;&eventValue=(?<eventValue>.*?)&;&eventAttribute=(?<eventAttribute>[^ ]*)\s*(?<httpprotocol>[^" ]*)[\s\W]*(?<httpcode>[^ ]*)\s*(?<size>[^" ]*)[\s"]*(?<cityname>.*?)"\s"(?<regionname>.*?)"\s"(?<countryname>.*?)"\s"(?<referer>[^ ]*)\s*[\s^\W]*(?<useragent>.*)$/
    time_format %Y-%m-%d %H:%M:%S %z
</source>
<match track.access.log.**>
  type influxdb
  host pippo
  port  8086
  dbname tracking
  user  pippo
  password  pippo
  use_ssl false
  time_precision s
  tag_keys ["host", "date", "httprequest", "request", "env", "url", "pagetype", "referrer", "locale", "web_id", "hitid", "maid", "currency", "session", "countryId", "cityId", "venueId", "eventsId", "cart", "transactionId", "transactionTotal", "transactionMrgn", "transactionProducts", "customer", "eventName", "eventValue", "eventAttribute", "httpprotocol", "httpcode", "size", "cityname", "regionname", "countryname", "referer", "useragent"]
  sequence_tag _seq
  flush_interval 10
  retry_limit 0
</match>

This is the log received on the first source:

nginx: 10.0.4.140 88.181.150.136 - - [02/Mar/2016:18:41:19 +0000]  "GET /favicon.ico?random_1456909201 HTTP/1.1" "-" 200 5430 "https://www.musement.com/fr/barcelone/la-sagrada-familia-v/" "Amazon CloudFront" 0.000 - .

and this is the log received in the second:

62.19.79.55 - - 2016-03-02 19:42:11 +0100 "GET /track.gif?env=prod&&;&url=https%253A%252F%252Fwww.musement.com%252Fit%252Fvienna%252Fbiglietto-combinato-sissi-il-castello-di-schonbrunn-hofburg-e-il-museo-del-mobile-imperiale-2329%252F&;&page_type=frontend_event&;&referrer=https%253A%252F%252Fwww.google.it%252F&;&locale=it&;&id=5ed06fe6feeea5da65ce08f24335f2d7&;&hitid=5ed06fe6feeea5da65ce08f24335f2d7-1456944126582&;&maid=&;&agent=Mozilla/5.0%20(Windows%20NT%206.3;%20WOW64)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/48.0.2564.116%20Safari/537.36&;&currency=EUR&;&session=&;&countryId=10&;&cityId=82&;&venueId=&;&eventsId=2329&;&cart=&;&transactionId=&;&transactionTotal=&;&transactionMrgn=&;&transactionProducts=&;&customer=&;&eventName=&;&eventValue=&;&eventAttribute= HTTP/1.1" 200 6469 "-" "-" "IT" "https://www.musement.com/it/vienna/biglietto-combinato-sissi-il-castello-di-schonbrunn-hofburg-e-il-museo-del-mobile-imperiale-2329/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"

why?

[ Feature Request ] wildcards support for tag_keys

Practically, json format data will be flattened before sending to influxdb ,
example:

//  product log
{
    path: 'xxxxxxxx',

    product:
    {
        id: 'xxx',
        'name': 'xxx'
    }
    message: 'xxxx'
}

// task log
{
    path: 'xxxxxxxx',
    task:
    {
        id: 'xxx',
        user:
        {
            id: 'xxx',
            nickname: 'xxx'
        },
        product:
        {
            id: 'xxx',
            'name': 'xxx'
        }
    },
    message: 'xxxx'
}

logger.emit('info', flatten(obj))

if I want to mark all id as tags
I have to set tag_keys = [ 'product.id', 'task.user.id', 'task.product.id' ]

With more nested data structure, I have to mark more stupid chained key name manually

so could we add feature support wildcards
**.id as fluent <match> pattern
or regex format /$id/

???

how to prevent string auto transformed to number？

for example
A numeric No like '123456' don't want to mark as tag

`flush_interval` not being honored

Any reason the below fluentd configuration would have issues with flush_interval 1s. It seems to be using the default 60s flush_interval.

Example fluentd.conf

...
<match **>
  @type influxdb
  host  influxdb
  port  8086
  dbname test
  user  admin
  password  admin
  use_ssl false
  measurement pingdom
  flush_interval 1s
</match>

influxdb 0.9 compatibility?

It looks like this plugin isn't 0.9 compatible. I'm trying to use it against influxdb 0.9 and getting:

2015-06-04 15:07:04 -0700 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2015-06-04 15:07:05 -0700 error_class="InfluxDB::Error" error="404 page not found\n" plugin_id="object:3fd80a06ce04"

This looks a lot like this: influxdata/influxdb#2032

Assuming the problem is indeed 0.9 compatibility, it would be great to have a 0.9 compatible branch of this repository.

Problem with plugin updating influxdb

Im seeing the following error when trying to send dstat data through fleuntd to influxdb using this plugin.

Versions:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-dstat-0.2.5
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-influxdb-0.1.2
influxDB ( Tested version 0.6.0 & 0.8.2 )

The error is :

2014-09-14 16:17:28 -0400 [warn]: fluent/output.rb:344:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2014-09-14 16:17:28 -0400 error_class="Net::HTTPServerException" error="400 "Bad Request"" instance=-608640758
2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in error!' 2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:128:invalue'
2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-influxdb-0.1.2/lib/fluent/plugin/out_influxdb.rb:50:in write' 2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/buffer.rb:296:inwrite_chunk'
2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/buffer.rb:276:in pop' 2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/output.rb:311:intry_flush'
2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/output.rb:132:in `run'

---

My system configuration file for fluentd

type dstat tag input.dstat option -c delay 5

#option -cdmsy

type influxdb
host 192.168.1.6
port 8086
dbname dstat
user xxxx
password xxxx
time_precision s

---

Wireshark packet trace
Wireshare Packet Trace:

POST /db/dstat/series?u=xxxx&p=xxxx&time_precision=s HTTP/1.1
Content-Type: application/json; charset=utf-8
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: /
User-Agent: Ruby
Connection: close
Host: 192.168.1.6:8086
Content-Length: 2571

[{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"1.148","sys":"0.863","idl":"97.556","wai":"0.135","hiq":"0.049","siq":"0.249"}},1410673009]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"3.441","sys":"0.810","idl":"95.749","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673009]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.202","sys":"0.202","idl":"99.194","wai":"0.0","hiq":"0.0","siq":"0.403"}},1410673014]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.202","idl":"99.798","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673019]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.202","sys":"0.0","idl":"99.798","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673024]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.202","idl":"99.597","wai":"0.0","hiq":"0.202","siq":"0.0"}},1410673029]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.0","idl":"100.0","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673034]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.203","idl":"99.797","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673039]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.202","idl":"99.798","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673044]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.0","idl":"100.0","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673049]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.0","idl":"100.0","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673054]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.201","sys":"0.201","idl":"99.598","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673059]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.0","idl":"99.798","wai":"0.0","hiq":"0.202","siq":"0.0"}},1410673064]]}]

====>
HTTP/1.1 400 Bad Request
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Access-Control-Allow-Methods: GET, POST, PUT, DELETE
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 2592000
Content-Type: text/plain
Date: Sun, 14 Sep 2014 05:37:45 GMT
Content-Length: 36
Connection: close

Unknown type map[string]interface {}

Handle InfluxDB::Errors Properly

I found an issue where the fluent-plugin-influxdb does not handle fatal influxdb errors well. I was switching from calling influxdb direct and using fluent so we had the time field set, but this plugin tried to set it too. This caused influxdb to throw an error:

2015-06-05 01:21:38 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-06-05 01:21:38 +0000 error_class="InfluxDB::Error" error="Cannot have duplicate field names" plugin_id="object:3fcef1e5c70c"
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:234:in `block in post'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:264:in `call'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:264:in `connect_with_retry'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:227:in `post'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:170:in `_write'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:163:in `write_point'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-influxdb-0.1.4/lib/fluent/plugin/out_influxdb.rb:52:in `block in write'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_memory.rb:61:in `feed_each'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_memory.rb:61:in `msgpack_each'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-influxdb-0.1.4/lib/fluent/plugin/out_influxdb.rb:49:in `write'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:325:in `write_chunk'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:304:in `pop'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:321:in `try_flush'
  2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:140:in `run'
2015-06-05 01:21:39 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-06-05 01:21:40 +0000 error_class="InfluxDB::Error" error="Cannot have duplicate field names" plugin_id="object:3fcef1e5c70c"

When this happens, the plugin gets into a retry loop state that prohibits any new messages from being relayed correctly. Instead of trying to resend the buffer, these types of messages should be logged and discarded. This will allow valid messages to continue being logged.

Setting 'tag_keys' never happend for strings?

This chunk of code looks buggy.

fluent-plugin-influxdb/lib/fluent/plugin/out_influxdb.rb

Lines 113 to 120 in ab91b7e

	if tag_keys.empty? && !@auto_tags
	values = record
	tags = {}
	else
	values = {}
	tags = {}
	record.each_pair do |k, v|
	if (@auto_tags && v.is_a?(String)) || @tag_keys.include?(k)

'tag_keys' never works for strings or/and 'auto_tags' is false, it's right logic?

InfluxDB empty field if tag is used

Hi,

if I use tag_keys ["vhost_Uid"], series are created like something.metrics,vhost_Uid=123, but selecting from something.metrics, that vhost_Uid field is null. Is it really need to be skipped in values?

https://github.com/fangli/fluent-plugin-influxdb/blob/master/lib/fluent/plugin/out_influxdb.rb#L117-L122

Donatas.

Set timestamp with js new Date().getTime() ms will meet timestamp format error

config:
<match .influxdb.test.health>
@type influxdb
measurement: health
host 127.0.0.1
port 40002
dbname test
auto_tags true
tag_keys: ["key1", "key2"]
time_key time
time_precision ms

plugin will auto convert js "new Date().getTime()" integer value to a float, I guess it's an issue for ruby integer scope, I convert "new Date().getTime()" to a string as a work around.

comments1:
I think all timestamp must be an integer, so could you add an auto integer convert for timestamp.

error="undefined method `to_msgpack'

Eventhough fluent/fluent-plugin-mongo#79 was fixed, the error still occurs in OUTPUT influxdb plugin.

  2016-08-18 18:49:51 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-influxdb-0.2.8/lib/fluent/plugin/out_influxdb.rb:95:in `to_msgpack'

Connecting to database : no timeout ?

Hi,
I am facing an issue when fluentd is starting and tries to connect to an unavailable InfluxDB.
Last trace from log is "Connecting to database".

If the DB is unavailable I would like fluentd to open its http source and store items in disk buffer.

Is that possible ?

2018-12-22 01:45:55 +0800 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2018-12-22 01:45:55 +0800 [info]: using configuration file: <ROOT>
  <system>
    log_level debug
  </system>
  <source>
    @type http
    @id input_http
    port 8888
  </source>
  <filter mkd.**>
    @type record_transformer
    remove_keys _time
    <record>
      time ${record["_time"]}
    </record>
  </filter>
  <match mkd.dev.depth>
    @type influxdb
    host "analytics.local"
    port 8087
    dbname "dev_mkd"
    measurement "depth"
    tag_keys ["e","s","d","t"]
    time_precision "ns"
    <buffer>
      @type "file"
      path "/var/log/td-agent/buffer/mkd.dev.depth"
      flush_interval 10s
      queued_chunks_limit_size 100
    </buffer>
  </match>
  <match mkd.**>
    @type stdout
    @id output_stdout
  </match>
</ROOT>
2018-12-22 01:45:55 +0800 [info]: starting fluentd-1.2.6 pid=22069 ruby="2.4.4"
2018-12-22 01:45:55 +0800 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/embedded/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-elasticsearch' version '2.11.11'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-influxdb' version '1.0.0'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-kafka' version '0.7.9'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-record-modifier' version '1.1.0'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.1.0'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-s3' version '1.1.6'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-td' version '1.0.0'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-webhdfs' version '1.2.3'
2018-12-22 01:45:55 +0800 [info]: gem 'fluentd' version '1.2.6'
2018-12-22 01:45:55 +0800 [info]: adding filter pattern="mkd.**" type="record_transformer"
2018-12-22 01:45:55 +0800 [info]: adding match pattern="mkd.dev.depth" type="influxdb"
2018-12-22 01:45:55 +0800 [info]: adding match pattern="mkd.**" type="stdout"
2018-12-22 01:45:55 +0800 [info]: adding source type="http"
2018-12-22 01:45:55 +0800 [info]: #0 starting fluentd worker pid=22086 ppid=22081 worker=0
2018-12-22 01:45:55 +0800 [debug]: #0 buffer started instance=70077661079240 stage_size=1275 queue_size=0
2018-12-22 01:45:55 +0800 [info]: #0 Connecting to database: dev_mkd, host: analytics.local, port: 8087, username: root, use_ssl = false, verify_ssl = true

Multiple bugs report

1. if record contains a NULL as value, no Error, has POST log, no records, just gone

2. timeStamp type conversion Error, should be INTEGER or a DATE? but FLOAT with an Error now

3. auto_tags almost not work, in most time, cause fail.

tag keys broken?

/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/basic_parser.rb:88:in `parse_error!': got incomplete JSON array configuration at mqtt.conf line 18,9 (Fluent::ConfigParseError)
 17:   tag_keys ['key1', 'key2']
 18: </match>

     ---------^
    from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/literal_parser.rb:236:in `scan_json'
    from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/literal_parser.rb:57:in `parse_literal'
    from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/v1_parser.rb:128:in `parse_element'
    from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/v1_parser.rb:93:in `parse_element'
    from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/v1_parser.rb:41:in `parse!'
    from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/v1_parser.rb:31:in `parse'
    from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config.rb:29:in `parse'
    from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:412:in `read_config'
    from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:127:in `start'
    from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/command/fluentd.rb:167:in `<top (required)>'
    from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
    from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
    from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluentd:6:in `<top (required)>'
    from /opt/td-agent/embedded/bin/fluentd:23:in `load'
    from /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
    from /usr/sbin/td-agent:7:in `load'
    from /usr/sbin/td-agent:7:in `<main>'

Buffers are not flushed after td-agent restart

After td-agent restart these are not flushed to InfluxDB and InfluxDB returns response with code 400 (Bad Request). Removing these buffer's files and restarting td-agent, everything is fine, InfluxDB returns 204.

[~]# ls -la /var/log/td-agent/buffer/
total 163852
drwxr-xr-x 2 td-agent td-agent    4096 Nov 17 14:29 .
drwxr-xr-x 3 td-agent td-agent    4096 Nov 17 03:20 ..
-rw-r--r-- 1 td-agent td-agent 6204727 Nov 17 14:36 openresty.buffer..b541800455821e280.log
-rw-r--r-- 1 td-agent td-agent   68346 Nov 17 11:47 openresty.buffer..q5417dc17eb4c3ac2.log
-rw-r--r-- 1 td-agent td-agent 8388194 Nov 17 11:55 openresty.buffer..q5417dc1cb04ba5c3.log
-rw-r--r-- 1 td-agent td-agent 8388383 Nov 17 12:02 openresty.buffer..q5417ddee752d0acd.log
-rw-r--r-- 1 td-agent td-agent 8388508 Nov 17 12:11 openresty.buffer..q5417df82b373f84c.log
-rw-r--r-- 1 td-agent td-agent 8388184 Nov 17 12:21 openresty.buffer..q5417e17983be7504.log
-rw-r--r-- 1 td-agent td-agent 8387911 Nov 17 12:30 openresty.buffer..q5417e3b708d850da.log
-rw-r--r-- 1 td-agent td-agent 8388467 Nov 17 12:40 openresty.buffer..q5417e5c5915f2753.log
-rw-r--r-- 1 td-agent td-agent 8387715 Nov 17 12:49 openresty.buffer..q5417e7ea0f3f354b.log
-rw-r--r-- 1 td-agent td-agent 8387881 Nov 17 12:57 openresty.buffer..q5417e9e221e86eb0.log
-rw-r--r-- 1 td-agent td-agent 8388236 Nov 17 13:06 openresty.buffer..q5417ebc5ac077d42.log
-rw-r--r-- 1 td-agent td-agent 8388493 Nov 17 13:15 openresty.buffer..q5417edd2c3c807b6.log
-rw-r--r-- 1 td-agent td-agent 8388199 Nov 17 13:24 openresty.buffer..q5417efca0eb5e84f.log
-rw-r--r-- 1 td-agent td-agent 8388511 Nov 17 13:34 openresty.buffer..q5417f1b799fed6ab.log
-rw-r--r-- 1 td-agent td-agent 8388219 Nov 17 13:43 openresty.buffer..q5417f3e4ecafe8e4.log
-rw-r--r-- 1 td-agent td-agent 8388417 Nov 17 13:49 openresty.buffer..q5417f5f0e5c500cf.log
-rw-r--r-- 1 td-agent td-agent 8388300 Nov 17 13:57 openresty.buffer..q5417f759ad405209.log
-rw-r--r-- 1 td-agent td-agent 8388106 Nov 17 14:05 openresty.buffer..q5417f93bb3a68ac4.log
-rw-r--r-- 1 td-agent td-agent 8388072 Nov 17 14:12 openresty.buffer..q5417faf8a4073910.log
-rw-r--r-- 1 td-agent td-agent 8388408 Nov 17 14:19 openresty.buffer..q5417fc9077170e98.log
-rw-r--r-- 1 td-agent td-agent 8388289 Nov 17 14:29 openresty.buffer..q5417fe1919044b32.log

is this known bug or something is bad with configuration?

<match metrics.openresty>
  type influxdb
  host X.X.X.X
  port 8086
  user metrics
  password xxx
  dbname metrics
  time_key msec
  time_precision ms
  buffer_type file
  buffer_path /var/log/td-agent/buffer/openresty.buffer
  buffer_queue_limit 512
  flush_interval 5s
</match>

Slow writes into influx

I've been trying to troubleshoot an issue where writes into influx seem to come in very slowly over time. My setup is as follows:

influxdb 0.8.8
fluent-plugin-influxdb (0.1.4)
fluentd (0.12.8)

fluentd config:
<match fuse.**>
type influxdb
host ***
port 8086
dbname ***
user ***
password ***
time_precision ms
buffer_type memory
buffer_chunk_limit 524288 # 512 * 1024
buffer_queue_limit 1024
flush_interval 30
retry_limit 17
retry_wait 1.0

I've played around and tried a number of different combinations of the config.

I have 2 fluentd instances dumping into a single influxdb instance. All machines are barely registering any CPU usage when messages are being sent, but it takes 20-30 minutes for the buffers to clear out into influx after all the data has been written to fluentd. This is around 72000 messages total (each message is ~100 bytes) being sent.

Turning debug logging on in influx I see a ton of spam about authenticating the user specified in the fluentd config, 10-20 a second. Any thoughts on what the issue might be?

Ruby Version Problem

Dear Friends
I want to use your plugin but I get error.
ERROR: Error installing fluent-plugin-influxdb:
influxdb requires Ruby version >= 2.2.0.

I uninstalled my ruby and installed different versions from 2.0.0, 2.2.0, 2.3, 2.4
but the error still exist. would you please help me how to solve this.
Thank you.

Use of tag_parts from fluent-mixin-rewrite-tag-name

I would like to be able to use the fluentd tags for configuring this plugin, using the tag_parts mixin. For example, with this config extract, the database name is determine by the second part of the tag. When I try this, the dbname is kept as "${tag_parts[1]}"

<match influxdb.**>
    @type influxdb
    dbname ${tag_parts[1]}
    host influxdb
  </match>

From what google tells me, this mixin would be needed: https://github.com/y-ken/fluent-mixin-rewrite-tag-name.

Do you think this would be possible?

update rubygems to the latest version

Mark.
Will be done when I back to office.

Using nanosecond precision instead of sequence_tag

This is just a suggestion for discussion/consideration.

Currently the plugin increments sequence_tag when multiple entries with identical timestamp occur, and resets it to zero when an event with different timestamp is seen. There are two problems with this.

1. You could end up generating a lot of time series. Suppose, for example, that in one particular second you have an unusual burst of 1000 messages. You'll end up creating 1000 time series, and in future all queries will have to be across all those 1000 series, even though mostly they will be empty.

2. If the records are being written with timestamps that are not increasingly linearly - e.g. there is some "backfill" going on - then there is a high risk of overwriting previous data, because the sequence tag is reset to zero every time the source timestamp changes. This is mainly a problem when the source timestamp has only 1-second resolution.

I would like to suggest another approach, which is to make use of the nanosecond timestamp precision of influxdb.

There are several approaches to this, but I propose the following one:

• Send all timestamps to influxdb in nanoseconds - which is influxdb default anyway ¹
• Keep a counter which starts at a random value between 0 and 999999, and increments for every message. Never reset to zero, except to wrap around from 999999 to 0.
• Add this value to the nanosecond timestamp sent to influxdb

This means that the stored timestamps are at worst in error by one millisecond. The chances of conflict are extremely low. If your input records have 1-second resolution then either you would have to have one million events in a single second, or you would have to be backfilling to a previous second and be unlucky enough to hit the same range of timestamps.

Even if your input records have millisecond resolution, as long as they arrive with monotonically increasing timestamps there should not be a problem, although some reordering is possible for records in adjacent milliseconds when the counter wraps. Maybe when the time precision is millisecond or better, the counter should only run between 0 and 999, so the error is no worse than one microsecond. (This upper limit can be made configurable anyway)

I did consider some other options - e.g. generating a random offset between 0 and 999999 for each record, or using a hash of the record. The former has a higher probability of collision (around 50% probability of at least one collision when 1000 records are stored in a single second). The latter will not record repeated records at all, which is undesirable for logs where you may actually want to count multiple instances of the same event.

---

Footnotes

1. If time_precision rounding is required then do it in the plugin and convert back to nanoseconds; but I don't really see why anyone would want to reduce the resolution of their existing timestamps. ↩

accept integer for float field

Now, insert an integer to a float field will throw an "already exists as type float" error

because there isn't precision float in Javascript, e.x: 1.0, 2.0... 10.0 all will be converted to integer automatically.
And as described in the 3032, now all numbers in influxdb are float, so could this be fixed?
Or could add an auto parse func from String?