Filter plugin for modifying each event record for Fluentd
Adding arbitary field to event record without customizing existence plugin.
For example, generated event from in_tail doesn't contain "hostname" of running machine. In this case, you can use record_modifier to add "hostname" field to event record.
Use RubyGems:
gem install fluent-plugin-record-modifier
Use record_modifier
filter.
<filter pattern>
@type record_modifier
<record>
gen_host "#{Socket.gethostname}"
foo bar
</record>
</filter>
If following record is passed:
{"message":"hello world!"}
then you got new record like below:
{"message":"hello world!", "gen_host":"oreore-mac.local", "foo":"bar"}
You can also use record_transformer
like ${xxx}
placeholders and access tag
, time
, record
and tag_parts
values by Ruby code.
<filter pattern>
@type record_modifier
<record>
tag ${tag}
tag_extract ${tag_parts[0]}-${tag_pars[1]}-foo
formatted_time ${Time.at(time).to_s}
new_field foo:${record['key1'] + record['dict']['key']}
</record>
</filter>
record_modifier
is faster than record_transformer
. See this comment.
But unlike record_transformer
, record_modifier
doesn't support following features for now.
- tag_suffix and tag_prefix
- dynamic key placeholder
Fluentd including some plugins treats logs as a BINARY by default to forward. But a user sometimes wants to process logs depends on their requirements, e.g. handling char encoding correctly.
char_encoding
parameter is useful for this case.
<filter pattern>
@type record_modifier
# set UTF-8 encoding information to string.
char_encoding utf-8
# change char encoding from 'UTF-8' to 'EUC-JP'
char_encoding utf-8:euc-jp
</filter>
In char_encoding from:to
case, it replaces invalid character with safe character.
The logs include needless record keys in some cases.
You can remove it by using remove_keys
parameter.
<filter pattern>
@type record_modifier
# remove key1 and key2 keys from record
remove_keys key1,key2
</filter>
If following record is passed:
{"key1":"hoge", "key2":"foo", "key3":"bar"}
then you got new record like below:
{"key3":"bar"}
If you want to handle the set of explicitly specified keys, you can use whitelist_keys
of this plugin. It's exclusive with remove_keys
.
<filter pattern>
@type record_modifier
# remove all keys except for key1 and key2
whitelist_keys key1,key2
</filter>
If following record is passed:
{"key1":"hoge", "key2":"foo", "key3":"bar"}
then you got new record like below:
{"key1":"hoge", "key2":"foo"}
In v0.10, you can use record_modifier
output to emulate filter. record_modifier
output doesn't support <record>
way.
<match pattern>
type record_modifier
tag foo.filtered
gen_host "#{Socket.gethostname}"
foo bar
</match>
-
Adding following features if needed
- Replace record value
Author | Masahiro Nakagawa |
Copyright | Copyright (c) 2013- Masahiro Nakagawa |
License | MIT License |