Comments (8)
Use types
parameter in in_http: http://docs.fluentd.org/articles/parser-plugin-overview
from fluent-plugin-influxdb.
Is there any other way to insert integer or float to InfluxDb? I use in_forward input plugin, because I need to read data from docker. And it seems that you have to use docker with in_forward only or am I wrong? This option doesn't work too:
<filter **>
type record_transformer
enable_ruby
<record>
value ${record["value"].to_f}
</record>
</filter>
from fluent-plugin-influxdb.
@o1da Use auto_typecase
: http://docs.fluentd.org/articles/filter_record_transformer#autotypecast-optional
from fluent-plugin-influxdb.
@repeatedly Thank you for tip, but I still have same problem with it. My filter looks like this:
<filter **>
type record_transformer
renew_record
enable_ruby
auto_typecast
<record>
key ${/\?key=([^ ]*)/.match(record["log"]){|m| m[1]}}
value ${/\[0m([^ ]*) ms/.match(record["log"]){|m| m[1]}}
</record>
</filter>
Messages which I'm parsing looks like #033[0mGET /retro/7/61/[email protected]?key=D82edpaWqKQaiQSNtXrr #033[32m200 #033[0m0.708 ms - 18660#033[0m
and I get key "D82edpaWqKQaiQSNtXrr"
which is OK, but value is "0.708"
which is not ok, it should be 0.708
and it is the same if I add another filter:
<filter **>
type record_transformer
auto_typecast
value ${value}
</filter>
What am I doing wrong?
from fluent-plugin-influxdb.
value ${/[0m([^ ]*) ms/.match(record["log"]){|m| m[1]}}
Why don't you call to_f
like value ${record["value"].to_f}
?
from fluent-plugin-influxdb.
I solved it with second filter.
<filter **>
type record_transformer
auto_typecast
enable_ruby
<record>
value ${record["value"].to_f}
</record>
</filter>
I overlooked info about limitation for single placeholder in auto_typecast
before.
@repeatedly Thank you very much :-)
from fluent-plugin-influxdb.
I solved it with second filter.
So you setup record_transformer
filter chain?
On my environment, ${/\[0m([^ ]*) ms/.match(record["log"]){|m| m[1].to_f}}
works.
% fluentd -c o1da_case.conf
2016-05-03 09:07:35 +0900 [info]: reading config file path="o1da_case.conf"
2016-05-03 09:07:35 +0900 [info]: starting fluentd-0.12.22
2016-05-03 09:07:35 +0900 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.0.4'
2016-05-03 09:07:35 +0900 [info]: gem 'fluentd' version '0.12.22'
2016-05-03 09:07:35 +0900 [info]: adding filter pattern="**" type="record_transformer"
2016-05-03 09:07:35 +0900 [info]: adding match pattern="**" type="stdout"
2016-05-03 09:07:35 +0900 [info]: adding source type="forward"
2016-05-03 09:07:35 +0900 [info]: using configuration file: <ROOT>
<source>
@type forward
</source>
<filter **>
type record_transformer
renew_record
enable_ruby
auto_typecast
<record>
key ${/\?key=([^ ]*)/.match(record["log"]){|m| m[1]}}
value ${/\[0m([^ ]*) ms/.match(record["log"]){|m| m[1].to_f}}
</record>
</filter>
<match **>
@type stdout
</match>
</ROOT>
2016-05-03 09:07:35 +0900 [info]: listening fluent socket on 0.0.0.0:24224
2016-05-03 09:07:38 +0900 test.foo: {"key":"D82edpaWqKQaiQSNtXrr","value":0.708}
from fluent-plugin-influxdb.
This configuration doesn't work, but I have different version and another plugins and installed td-agent package from debian jessie repo - maybe it makes the difference.
2016-05-20 12:42:39 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2016-05-20 12:42:39 +0000 [info]: starting fluentd-0.12.20
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-influxdb' version '0.2.7'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.12'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-s3' version '0.6.5'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2016-05-20 12:42:39 +0000 [info]: gem 'fluentd' version '0.12.20'
2016-05-20 12:42:39 +0000 [info]: adding filter pattern="**" type="record_transformer"
2016-05-20 12:42:39 +0000 [info]: adding match pattern="**" type="stdout"
2016-05-20 12:42:39 +0000 [info]: adding source type="forward"
2016-05-20 12:42:39 +0000 [info]: using configuration file: <ROOT>
<source>
type forward
heartbeat_type tcp
port 24224
bind 127.0.0.1
</source>
<filter **>
type record_transformer
renew_record
enable_ruby
auto_typecast
<record>
key ${/\?key=([^ ]*)/.match(record["log"]){|m| m[1]}}
value ${/\[0m([^ ]*) ms/.match(record["log"]){|m| m[1].to_f}}
</record>
</filter>
<match **>
@type stdout
</match>
</ROOT>
2016-05-20 12:42:39 +0000 [warn]: parameter 'heartbeat_type' in <source>
type forward
heartbeat_type tcp
port 24224
bind 127.0.0.1
</source> is not used.
2016-05-20 12:42:39 +0000 [info]: listening fluent socket on 127.0.0.1:24224
2016-05-20 14:23:18 +0000 test: {"key":"D82edpaWqKQaiQSNtXrr","value":"70.422"}
Only configuration with second filter works and it is OK for me I don't need to tweak it any more.
2016-05-20 14:28:49 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2016-05-20 14:28:49 +0000 [info]: starting fluentd-0.12.20
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-influxdb' version '0.2.7'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.12'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-s3' version '0.6.5'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2016-05-20 14:28:49 +0000 [info]: gem 'fluentd' version '0.12.20'
2016-05-20 14:28:49 +0000 [info]: adding filter pattern="**" type="record_transformer"
2016-05-20 14:28:49 +0000 [info]: adding filter pattern="**" type="record_transformer"
2016-05-20 14:28:49 +0000 [info]: adding match pattern="**" type="stdout"
2016-05-20 14:28:49 +0000 [info]: adding source type="forward"
2016-05-20 14:28:49 +0000 [info]: using configuration file: <ROOT>
<source>
type forward
heartbeat_type tcp
port 24224
bind 127.0.0.1
</source>
<filter **>
type record_transformer
renew_record
enable_ruby
<record>
key ${/\?key=([^ ]*)/.match(record["log"]){|m| m[1]}}
value ${/\[0m([^ ]*) ms/.match(record["log"]){|m| m[1]}}
</record>
</filter>
<filter **>
type record_transformer
auto_typecast
enable_ruby
<record>
value ${record["value"].to_f}
</record>
</filter>
<match **>
@type stdout
</match>
</ROOT>
2016-05-20 14:28:49 +0000 [warn]: parameter 'heartbeat_type' in <source>
type forward
heartbeat_type tcp
port 24224
bind 127.0.0.1
</source> is not used.
2016-05-20 14:28:49 +0000 [info]: listening fluent socket on 127.0.0.1:24224
2016-05-20 14:29:11 +0000 test: {"key":"D82edpaWqKQaiQSNtXrr","value":91.37}
from fluent-plugin-influxdb.
Related Issues (20)
- auto_tag functionality
- Number/float field are pushed into influxdb as string HOT 1
- Adding support for InfluxDB 2? HOT 2
- Dynamic measurement value
- How do I inject custom key/values
- Errors transporting Collectd metrics into Influxdb via fluentbit
- retry parameter not working at all
- plugin makes fluentd fail having fluentd 1.15.x and ruby 3 HOT 2
- Unauthorized Error
- Setting 'tag_keys' never happend for strings?
- Set timestamp with js new Date().getTime() ms will meet timestamp format error HOT 3
- Using nanosecond precision instead of sequence_tag HOT 1
- Multiple bugs report HOT 9
- Feature Request: Option to write Boolean as Integer
- accept integer for float field HOT 5
- Connecting to database : no timeout ?
- Allow empty strings for fields
- memory leak if influxdb goes down HOT 3
- [ Feature Request ] wildcards support for tag_keys
- how to prevent string auto transformed to number?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-plugin-influxdb.