custom timestamp format - millisecs are down to 0 about fluent-plugin-elasticsearch

[superdump]

A solution for 0.12.x could be:

<filter **>
  @type record_transformer
  enable_ruby
  <record>
    timestamp ${ require 'time'; Time.now.utc.iso8601(3) }
  </record>
</filter>

And then use time_key timestamp in the elasticsearch plugin config.

[cb372]

I have exactly the same problem.

I have index templates set up for ES, so as a workaround I'll try adding a mapping for the @timestamp column with format dateTime and see if that fixes it.

[cb372]

Follow-up: no, that didn't work at all. It looks like the timestamp that fluent-elasticsearch-plugin doesn't comply with the dateTime format, so the insert to ES fails.

[tmeinlschmidt]

I've temporarily resolved adding another column with desider accuracy and using as a sorting point

[shivaken]

Hi, I tried to put timestamp with millisec.
if '@timestamp' is formatted by strftime("%Y-%m-%dT%H:%M:%S.%L%z") like '2014-11-14T15:37:04.731+0000', the timestamp is stored successfully and the date can be seen on Kibana.
Is it possible for fluent-plugin-elasticsearch to have a feature to combine the time valuable which is unixtime and additional millisecond data (if the record has) into this format?

[superdump]

It seems that this has been overhauled in fluentd 0.14.0pre1 and timestamps with nanosecond accuracy are supported.

In any case, can someone provide exact information here about how to configure fluentd and fluent-plugin-elasticsearch with 0.12.x to write milliseconds into the @timestamp field?

[tobilarscheid]

Hi @superdump and everyone else,

It seems that this has been overhauled in fluentd 0.14.0pre1 and timestamps with nanosecond accuracy are supported.

does this mean using fluentd 0.14 with this plugin would give me nano or at least millisecond precision? Are any changes in fluent-plugin-elasticsearch required?

[crackmac]

I believe #223 will handle the nano sec issue once it is merged in. It look like some failed testing is all that is holding it back.