Comments (9)
macdjord
commented on May 24, 2024
Update: added 'log_level trace' to conf, reran:
2016-08-18 11:30:49 -0400 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2016-08-18 11:30:49 -0400 [info]: starting fluentd-0.12.26
2016-08-18 11:30:49 -0400 [trace]: registered buffer plugin 'file'
2016-08-18 11:30:49 -0400 [trace]: registered buffer plugin 'memory'
2016-08-18 11:30:49 -0400 [trace]: registered filter plugin 'grep'
2016-08-18 11:30:49 -0400 [trace]: registered filter plugin 'record_transformer'
2016-08-18 11:30:49 -0400 [trace]: registered filter plugin 'stdout'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'debug_agent'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'dummy'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'exec'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'forward'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'gc_stat'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'http'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'monitor_agent'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'object_space'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'status'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'unix'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'syslog'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'tail'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'tcp'
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'udp'
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'copy'
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'exec'
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'exec_filter'
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'file'
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'forward'
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'null'
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'relabel'
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'roundrobin'
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'stdout'
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'tcp'
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'unix'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-concat' version '0.5.0'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-forest' version '0.3.1'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-mongo' version '0.7.13'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-multi-format-parser' version '0.0.2'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-s3' version '0.6.8'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-sar' version '0.0.4'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-secure-forward' version '0.4.3'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-systemd' version '0.0.3'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-tail-multiline' version '0.1.5'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-08-18 11:30:49 -0400 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2'
2016-08-18 11:30:49 -0400 [info]: gem 'fluentd' version '0.12.26'
2016-08-18 11:30:49 -0400 [info]: gem 'fluentd' version '0.10.61'
2016-08-18 11:30:49 -0400 [info]: adding filter in @DEFAULT pattern="**" type="stdout"
2016-08-18 11:30:49 -0400 [info]: adding match in @DEFAULT pattern="**" type="copy"
2016-08-18 11:30:49 -0400 [debug]: adding store type="secure_forward"
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'secure_forward'
2016-08-18 11:30:49 -0400 [debug]: adding store type="forest"
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'forest'
2016-08-18 11:30:49 -0400 [info]: adding filter pattern="**" type="record_transformer"
2016-08-18 11:30:49 -0400 [info]: adding filter pattern="fluent.{fatal,error,warn,info,debug,trace}.**" type="record_transformer"
2016-08-18 11:30:49 -0400 [info]: adding filter pattern="docker-container.elasticsearch-soraka.**" type="concat"
2016-08-18 11:30:49 -0400 [trace]: registered filter plugin 'concat'
2016-08-18 11:30:49 -0400 [info]: adding filter pattern="docker-container.fluentd-soraka.**" type="concat"
2016-08-18 11:30:49 -0400 [info]: adding filter pattern="docker-container.nmap-scanner.**" type="concat"
2016-08-18 11:30:49 -0400 [info]: adding match pattern="filter.service.fluentd-forwarder.**" type="rewrite_tag_filter"
2016-08-18 11:30:49 -0400 [trace]: registered output plugin 'rewrite_tag_filter'
2016-08-18 11:30:49 -0400 [info]: adding rewrite_tag_filter rule: rewriterule1 ["log", /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} [-+]\d{4} \[\w+?\]: Timeout flush:/, "", "skip.${tag}"]
2016-08-18 11:30:49 -0400 [info]: adding rewrite_tag_filter rule: rewriterule2 ["log", /.*/, "", "${tag}"]
2016-08-18 11:30:49 -0400 [info]: adding match pattern="skip.service.fluentd-forwarder.**" type="rewrite_tag_filter"
2016-08-18 11:30:49 -0400 [info]: adding rewrite_tag_filter rule: rewriterule1 ["log", /.*/, "", "${tag}"]
2016-08-18 11:30:49 -0400 [info]: adding filter pattern="service.fluentd-forwarder.**" type="concat"
2016-08-18 11:30:49 -0400 [info]: adding match pattern="**" type="relabel"
2016-08-18 11:30:49 -0400 [info]: adding source type="dummy"
2016-08-18 11:30:49 -0400 [info]: adding source type="forward"
2016-08-18 11:30:49 -0400 [info]: adding source type="systemd"
2016-08-18 11:30:49 -0400 [trace]: registered input plugin 'systemd'
2016-08-18 11:30:50 -0400 [info]: adding source type="tail"
2016-08-18 11:30:50 -0400 [info]: using configuration file: <ROOT>
<system>
log_level trace
</system>
<source>
@type dummy
tag fluent.heartbeat.buffered
rate 1
dummy {"message":"heartbeat","level":"debug"}
</source>
<source>
@type forward
bind 0.0.0.0
port 24224
</source>
<source>
@type systemd
tag systemd.docker-engine
@log_level trace
path /var/log/journal/
filters [{"_SYSTEMD_UNIT":"docker.service"}]
strip_underscores true
pos_file /var/lib/td-agent/pos/docker-engine.pos
read_from_head true
</source>
<source>
@type tail
tag filter.service.fluentd-forwarder
path /var/log/td-agent/td-agent.log
pos_file /var/lib/td-agent/pos/td-agent.pos
read_from_head true
format /(?<log>^((?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} [-+]\d{4}) \[(?<level>\w+?)\])?.*$)/
keep_time_key false
</source>
<filter **>
@type record_transformer
<record>
hostname ${hostname}
</record>
</filter>
<filter fluent.{fatal,error,warn,info,debug,trace}.**>
@type record_transformer
<record>
level ${tag_parts[1]}
</record>
</filter>
<filter docker-container.elasticsearch-soraka.**>
@type concat
key log
stream_identity_key source
multiline_start_regexp /^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\]/
flush_interval 10
timeout_label @DEFAULT
</filter>
<filter docker-container.fluentd-soraka.**>
@type concat
key log
stream_identity_key source
multiline_start_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/
flush_interval 10
timeout_label @DEFAULT
</filter>
<filter docker-container.nmap-scanner.**>
@type concat
key log
stream_identity_key source
multiline_start_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}:/
flush_interval 120
timeout_label @DEFAULT
</filter>
<match filter.service.fluentd-forwarder.**>
@type rewrite_tag_filter
remove_tag_prefix filter
rewriterule1 log ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} [-+]\d{4} \[\w+?\]: Timeout flush: skip.${tag}
rewriterule2 log .* ${tag}
</match>
<match skip.service.fluentd-forwarder.**>
@type rewrite_tag_filter
@label @DEFAULT
remove_tag_prefix skip
rewriterule1 log .* ${tag}
</match>
<filter service.fluentd-forwarder.**>
@type concat
key log
multiline_start_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/
flush_interval 10
timeout_label @DEFAULT
</filter>
<match **>
@type relabel
@label @DEFAULT
</match>
<label @DEFAULT>
<filter **>
@type stdout
</filter>
<match **>
@type copy
<store>
@type secure_forward
buffer_type file
buffer_path /var/spool/td-agent/buf/fek.*.buffer
secure true
shared_key xxxxxx
self_hostname [REDACTED]
ca_cert_path /etc/td-agent/ssl/certs/ca_cert.pem
<server>
host 10.6.20.18
port 24284
</server>
</store>
<store>
@type forest
subtype copy
<case docker-container.**>
<store>
@type file
path /var/log/td-agent/docker-containers/${tag_parts[1..-1]}
time_slice_format %Y%m%d
time_slice_wait 10m
time_format %Y%m%d@%H%M%S%z
utc
compress gzip
</store>
</case>
</store>
</match>
</label>
</ROOT>
2016-08-18 11:30:50 -0400 [debug]: starting secure-forward
2016-08-18 11:30:50 -0400 [debug]: start to connect target nodes
2016-08-18 11:30:50 -0400 [debug]: connecting node host="10.6.20.18" port=24284
2016-08-18 11:30:50 -0400 [info]: listening fluent socket on 0.0.0.0:24224
2016-08-18 11:30:50 -0400 [debug]: starting client
2016-08-18 11:30:50 -0400 [debug]: create tcp socket to node host="10.6.20.18" address="10.6.20.18" port=24284
2016-08-18 11:30:50 -0400 [trace]: changing socket options
2016-08-18 11:30:50 -0400 [trace]: initializing SSL contexts
2016-08-18 11:30:50 -0400 [trace]: setting SSL verification options
2016-08-18 11:30:50 -0400 systemd.docker-engine: {"PRIORITY":"6","UID":"0","GID":"0","SYSTEMD_SLICE":"system.slice","BOOT_ID":"9bdaf4b9512f493c99a8be69ae697588","MACHINE_ID":"ab6857580005461cab71a95470ac18e6","HOSTNAME":"[REDACTED]","SYSLOG_FACILITY":"3","CAP_EFFECTIVE":"1fffffffff","TRANSPORT":"stdout","SYSLOG_IDENTIFIER":"docker","MESSAGE":"time=\"2016-07-28T10:24:03.361026938-04:00\" level=info msg=\"New containerd process, pid: 2089\\n\"","PID":"941","COMM":"docker","EXE":"/usr/bin/docker","CMDLINE":"/usr/bin/docker daemon -H fd://","SYSTEMD_CGROUP":"/system.slice/docker.service","SYSTEMD_UNIT":"docker.service","hostname":"[REDACTED]"}
2016-08-18 11:30:50 -0400 [info]: following tail of /var/log/td-agent/td-agent.log
2016-08-18 11:30:50 -0400 [trace]: set verify_mode VERIFY_PEER
2016-08-18 11:30:50 -0400 [info]: out_forest plants new output: copy for tag 'systemd.docker-engine'
2016-08-18 11:30:50 -0400 [trace]: set to use private CA path="/etc/td-agent/ssl/certs/ca_cert.pem"
2016-08-18 11:30:50 -0400 [debug]: trying to connect ssl session host="10.6.20.18" address="10.6.20.18" port=24284
2016-08-18 11:30:50 -0400 [trace]: connecting... host="10.6.20.18" address="10.6.20.18" port=24284
2016-08-18 11:30:51 -0400 [info]: process finished code=134
2016-08-18 11:30:51 -0400 [error]: fluentd main process died unexpectedly. restarting.
2016-08-18 11:30:51 -0400 [info]: starting fluentd-0.12.26
2016-08-18 11:30:51 -0400 [trace]: registered buffer plugin 'file'
2016-08-18 11:30:51 -0400 [trace]: registered buffer plugin 'memory'
2016-08-18 11:30:51 -0400 [trace]: registered filter plugin 'grep'
2016-08-18 11:30:51 -0400 [trace]: registered filter plugin 'record_transformer'
2016-08-18 11:30:51 -0400 [trace]: registered filter plugin 'stdout'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'debug_agent'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'dummy'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'exec'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'forward'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'gc_stat'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'http'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'monitor_agent'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'object_space'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'status'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'unix'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'syslog'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'tail'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'tcp'
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'udp'
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'copy'
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'exec'
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'exec_filter'
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'file'
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'forward'
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'null'
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'relabel'
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'roundrobin'
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'stdout'
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'tcp'
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'unix'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-concat' version '0.5.0'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-forest' version '0.3.1'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-mongo' version '0.7.13'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-multi-format-parser' version '0.0.2'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-s3' version '0.6.8'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-sar' version '0.0.4'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-secure-forward' version '0.4.3'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-systemd' version '0.0.3'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-tail-multiline' version '0.1.5'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-08-18 11:30:51 -0400 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2'
2016-08-18 11:30:51 -0400 [info]: gem 'fluentd' version '0.12.26'
2016-08-18 11:30:51 -0400 [info]: gem 'fluentd' version '0.10.61'
2016-08-18 11:30:51 -0400 [info]: adding filter in @DEFAULT pattern="**" type="stdout"
2016-08-18 11:30:51 -0400 [info]: adding match in @DEFAULT pattern="**" type="copy"
2016-08-18 11:30:51 -0400 [debug]: adding store type="secure_forward"
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'secure_forward'
2016-08-18 11:30:51 -0400 [debug]: adding store type="forest"
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'forest'
2016-08-18 11:30:51 -0400 [info]: adding filter pattern="**" type="record_transformer"
2016-08-18 11:30:51 -0400 [info]: adding filter pattern="fluent.{fatal,error,warn,info,debug,trace}.**" type="record_transformer"
2016-08-18 11:30:51 -0400 [info]: adding filter pattern="docker-container.elasticsearch-soraka.**" type="concat"
2016-08-18 11:30:51 -0400 [trace]: registered filter plugin 'concat'
2016-08-18 11:30:51 -0400 [info]: adding filter pattern="docker-container.fluentd-soraka.**" type="concat"
2016-08-18 11:30:51 -0400 [info]: adding filter pattern="docker-container.nmap-scanner.**" type="concat"
2016-08-18 11:30:51 -0400 [info]: adding match pattern="filter.service.fluentd-forwarder.**" type="rewrite_tag_filter"
2016-08-18 11:30:51 -0400 [trace]: registered output plugin 'rewrite_tag_filter'
2016-08-18 11:30:51 -0400 [info]: adding rewrite_tag_filter rule: rewriterule1 ["log", /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} [-+]\d{4} \[\w+?\]: Timeout flush:/, "", "skip.${tag}"]
2016-08-18 11:30:51 -0400 [info]: adding rewrite_tag_filter rule: rewriterule2 ["log", /.*/, "", "${tag}"]
2016-08-18 11:30:51 -0400 [info]: adding match pattern="skip.service.fluentd-forwarder.**" type="rewrite_tag_filter"
2016-08-18 11:30:51 -0400 [info]: adding rewrite_tag_filter rule: rewriterule1 ["log", /.*/, "", "${tag}"]
2016-08-18 11:30:51 -0400 [info]: adding filter pattern="service.fluentd-forwarder.**" type="concat"
2016-08-18 11:30:51 -0400 [info]: adding match pattern="**" type="relabel"
2016-08-18 11:30:51 -0400 [info]: adding source type="dummy"
2016-08-18 11:30:51 -0400 [info]: adding source type="forward"
2016-08-18 11:30:51 -0400 [info]: adding source type="systemd"
2016-08-18 11:30:51 -0400 [trace]: registered input plugin 'systemd'
2016-08-18 11:30:51 -0400 [info]: adding source type="tail"
2016-08-18 11:30:51 -0400 [info]: using configuration file: <ROOT>
<system>
log_level trace
</system>
<source>
@type dummy
tag fluent.heartbeat.buffered
rate 1
dummy {"message":"heartbeat","level":"debug"}
</source>
<source>
@type forward
bind 0.0.0.0
port 24224
</source>
<source>
@type systemd
tag systemd.docker-engine
@log_level trace
path /var/log/journal/
filters [{"_SYSTEMD_UNIT":"docker.service"}]
strip_underscores true
pos_file /var/lib/td-agent/pos/docker-engine.pos
read_from_head true
</source>
<source>
@type tail
tag filter.service.fluentd-forwarder
path /var/log/td-agent/td-agent.log
pos_file /var/lib/td-agent/pos/td-agent.pos
read_from_head true
format /(?<log>^((?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} [-+]\d{4}) \[(?<level>\w+?)\])?.*$)/
keep_time_key false
</source>
<filter **>
@type record_transformer
<record>
hostname ${hostname}
</record>
</filter>
<filter fluent.{fatal,error,warn,info,debug,trace}.**>
@type record_transformer
<record>
level ${tag_parts[1]}
</record>
</filter>
<filter docker-container.elasticsearch-soraka.**>
@type concat
key log
stream_identity_key source
multiline_start_regexp /^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\]/
flush_interval 10
timeout_label @DEFAULT
</filter>
<filter docker-container.fluentd-soraka.**>
@type concat
key log
stream_identity_key source
multiline_start_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/
flush_interval 10
timeout_label @DEFAULT
</filter>
<filter docker-container.nmap-scanner.**>
@type concat
key log
stream_identity_key source
multiline_start_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}:/
flush_interval 120
timeout_label @DEFAULT
</filter>
<match filter.service.fluentd-forwarder.**>
@type rewrite_tag_filter
remove_tag_prefix filter
rewriterule1 log ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} [-+]\d{4} \[\w+?\]: Timeout flush: skip.${tag}
rewriterule2 log .* ${tag}
</match>
<match skip.service.fluentd-forwarder.**>
@type rewrite_tag_filter
@label @DEFAULT
remove_tag_prefix skip
rewriterule1 log .* ${tag}
</match>
<filter service.fluentd-forwarder.**>
@type concat
key log
multiline_start_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/
flush_interval 10
timeout_label @DEFAULT
</filter>
<match **>
@type relabel
@label @DEFAULT
</match>
<label @DEFAULT>
<filter **>
@type stdout
</filter>
<match **>
@type copy
<store>
@type secure_forward
buffer_type file
buffer_path /var/spool/td-agent/buf/fek.*.buffer
secure true
shared_key xxxxxx
self_hostname [REDACTED]
ca_cert_path /etc/td-agent/ssl/certs/ca_cert.pem
<server>
host 10.6.20.18
port 24284
</server>
</store>
<store>
@type forest
subtype copy
<case docker-container.**>
<store>
@type file
path /var/log/td-agent/docker-containers/${tag_parts[1..-1]}
time_slice_format %Y%m%d
time_slice_wait 10m
time_format %Y%m%d@%H%M%S%z
utc
compress gzip
</store>
</case>
</store>
</match>
</label>
</ROOT>
2016-08-18 11:30:51 -0400 [debug]: starting secure-forward
2016-08-18 11:30:51 -0400 [debug]: start to connect target nodes
2016-08-18 11:30:51 -0400 [debug]: connecting node host="10.6.20.18" port=24284
2016-08-18 11:30:51 -0400 [info]: listening fluent socket on 0.0.0.0:24224
2016-08-18 11:30:51 -0400 [info]: following tail of /var/log/td-agent/td-agent.log
2016-08-18 11:30:51 -0400 [info]: out_forest plants new output: copy for tag 'service.fluentd-forwarder'
2016-08-18 11:30:51 -0400 [debug]: starting client
2016-08-18 11:30:51 -0400 service.fluentd-forwarder: {"log":"2016-08-18 11:30:49 -0400 [info]: reading config file path=\"/etc/td-agent/td-agent.conf\"","level":"info","hostname":"[REDACTED]"}
2016-08-18 11:30:51 -0400 systemd.docker-engine: {"PRIORITY":"6","UID":"0","GID":"0","SYSTEMD_SLICE":"system.slice","BOOT_ID":"9bdaf4b9512f493c99a8be69ae697588","MACHINE_ID":"ab6857580005461cab71a95470ac18e6","HOSTNAME":"[REDACTED]","SYSLOG_FACILITY":"3","CAP_EFFECTIVE":"1fffffffff","TRANSPORT":"stdout","SYSLOG_IDENTIFIER":"docker","MESSAGE":"time=\"2016-07-28T10:24:03.361026938-04:00\" level=info msg=\"New containerd process, pid: 2089\\n\"","PID":"941","COMM":"docker","EXE":"/usr/bin/docker","CMDLINE":"/usr/bin/docker daemon -H fd://","SYSTEMD_CGROUP":"/system.slice/docker.service","SYSTEMD_UNIT":"docker.service","hostname":"[REDACTED]"}
2016-08-18 11:30:51 -0400 [debug]: create tcp socket to node host="10.6.20.18" address="10.6.20.18" port=24284
2016-08-18 11:30:51 -0400 [trace]: changing socket options
2016-08-18 11:30:51 -0400 [trace]: initializing SSL contexts
2016-08-18 11:30:51 -0400 [trace]: setting SSL verification options
2016-08-18 11:30:51 -0400 [trace]: set verify_mode VERIFY_PEER
2016-08-18 11:30:51 -0400 service.fluentd-forwarder: {"log":"2016-08-18 11:30:49 -0400 [info]: starting fluentd-0.12.26","level":"info","hostname":"[REDACTED]"}
2016-08-18 11:30:51 -0400 [trace]: set to use private CA path="/etc/td-agent/ssl/certs/ca_cert.pem"
2016-08-18 11:30:51 -0400 [info]: out_forest plants new output: copy for tag 'systemd.docker-engine'
2016-08-18 11:30:51 -0400 [debug]: trying to connect ssl session host="10.6.20.18" address="10.6.20.18" port=24284
2016-08-18 11:30:51 -0400 [trace]: connecting... host="10.6.20.18" address="10.6.20.18" port=24284
2016-08-18 11:30:51 -0400 [info]: process finished code=134
2016-08-18 11:30:51 -0400 [warn]: process died within 1 second. exit.
from fluent-plugin-systemd.
errm
commented on May 24, 2024
What does your environment look like? Are you running fluentd on the same host as systemd, or inside a container?
Could you try isolating the problem by running fluentd with just the systemd input and a basic output plugin like file or something...
The verbose logs would be super useful also, if you could run fluentd with -vv http://docs.fluentd.org/articles/trouble-shooting#turn-on-verbose-logging
Sorry I would love to be more help, but I can't see anything in the logs that relates to this plugin...
from fluent-plugin-systemd.
macdjord
commented on May 24, 2024
Environment: CentOS 7, not containerized
Minimalist config, verbose logging on:
2016-08-19 16:25:00 -0400 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2016-08-19 16:25:00 -0400 [info]: starting fluentd-0.12.26
2016-08-19 16:25:00 -0400 [trace]: registered buffer plugin 'file'
2016-08-19 16:25:00 -0400 [trace]: registered buffer plugin 'memory'
2016-08-19 16:25:00 -0400 [trace]: registered filter plugin 'grep'
2016-08-19 16:25:00 -0400 [trace]: registered filter plugin 'record_transformer'
2016-08-19 16:25:00 -0400 [trace]: registered filter plugin 'stdout'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'debug_agent'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'dummy'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'exec'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'forward'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'gc_stat'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'http'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'monitor_agent'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'object_space'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'status'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'unix'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'syslog'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'tail'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'tcp'
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'udp'
2016-08-19 16:25:00 -0400 [trace]: registered output plugin 'copy'
2016-08-19 16:25:00 -0400 [trace]: registered output plugin 'exec'
2016-08-19 16:25:00 -0400 [trace]: registered output plugin 'exec_filter'
2016-08-19 16:25:00 -0400 [trace]: registered output plugin 'file'
2016-08-19 16:25:00 -0400 [trace]: registered output plugin 'forward'
2016-08-19 16:25:00 -0400 [trace]: registered output plugin 'null'
2016-08-19 16:25:00 -0400 [trace]: registered output plugin 'relabel'
2016-08-19 16:25:00 -0400 [trace]: registered output plugin 'roundrobin'
2016-08-19 16:25:00 -0400 [trace]: registered output plugin 'stdout'
2016-08-19 16:25:00 -0400 [trace]: registered output plugin 'tcp'
2016-08-19 16:25:00 -0400 [trace]: registered output plugin 'unix'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-concat' version '0.5.0'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-forest' version '0.3.1'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-mongo' version '0.7.13'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-multi-format-parser' version '0.0.2'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-s3' version '0.6.8'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-sar' version '0.0.4'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-secure-forward' version '0.4.3'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-systemd' version '0.0.3'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-tail-multiline' version '0.1.5'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-08-19 16:25:00 -0400 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2'
2016-08-19 16:25:00 -0400 [info]: gem 'fluentd' version '0.12.26'
2016-08-19 16:25:00 -0400 [info]: gem 'fluentd' version '0.10.61'
2016-08-19 16:25:00 -0400 [info]: adding filter pattern="**" type="stdout"
2016-08-19 16:25:00 -0400 [info]: adding match pattern="**" type="file"
2016-08-19 16:25:00 -0400 [info]: adding source type="systemd"
2016-08-19 16:25:00 -0400 [trace]: registered input plugin 'systemd'
2016-08-19 16:25:00 -0400 [info]: using configuration file: <ROOT>
<system>
log_level trace
</system>
<source>
@type systemd
tag systemd.docker-engine
path /var/log/journal/
filters [{"_SYSTEMD_UNIT":"docker.service"}]
strip_underscores true
pos_file /var/lib/td-agent/pos/docker-engine.pos
read_from_head true
</source>
<filter **>
@type stdout
</filter>
<match **>
@type file
path /var/log/td-agent/test
time_slice_format %Y%m%d
time_slice_wait 10m
time_format %Y%m%dT%H%M%S%z
utc
buffer_path /var/log/td-agent/test.*
</match>
</ROOT>
2016-08-19 16:25:00 -0400 systemd.docker-engine: {"PRIORITY":"6","UID":"0","GID":"0","SYSTEMD_SLICE":"system.slice","BOOT_ID":"9bdaf4b9512f493c99a8be69ae697588","MACHINE_ID":"ab6857580005461cab71a95470ac18e6","HOSTNAME":"ws-j-macdonald.indexexchange.com","SYSLOG_FACILITY":"3","CAP_EFFECTIVE":"1fffffffff","TRANSPORT":"stdout","SYSLOG_IDENTIFIER":"docker","MESSAGE":"time=\"2016-07-28T10:24:03.361026938-04:00\" level=info msg=\"New containerd process, pid: 2089\\n\"","PID":"941","COMM":"docker","EXE":"/usr/bin/docker","CMDLINE":"/usr/bin/docker daemon -H fd://","SYSTEMD_CGROUP":"/system.slice/docker.service","SYSTEMD_UNIT":"docker.service"}
2016-08-19 16:25:00 -0400 [info]: process finished code=134
2016-08-19 16:25:00 -0400 [warn]: process died within 1 second. exit.
The relevant files on disk:
[root@[REDACTED] td-agent]# ls -lr /var/log/journal/
total 8
drwxr-sr-x+ 2 root systemd-journal 4096 Aug 18 05:46 ab6857580005461cab71a95470ac18e6
[root@[REDACTED] td-agent]# ls -lR /var/log/journal/
/var/log/journal/:
total 8
drwxr-sr-x+ 2 root systemd-journal 4096 Aug 18 05:46 ab6857580005461cab71a95470ac18e6
/var/log/journal/ab6857580005461cab71a95470ac18e6:
total 221208
-rw-r-xr--+ 1 root root 58720256 Aug 16 18:17 [email protected]~
-rw-r-xr--+ 1 root root 125829120 Aug 10 15:32 system@a312b3bcde654716b893c1cdce0e70e9-0000000000000001-000538b2d08945b7.journal
-rw-r-x---+ 1 root systemd-journal 8388608 Aug 17 01:37 system@cf1b42928e73439dbba46bbd3d87b361-0000000000000001-00053a37b6531f9b.journal
-rw-r-----+ 1 root systemd-journal 8388608 Aug 17 15:33 system@cf1b42928e73439dbba46bbd3d87b361-0000000000001742-00053a3ddd8cb8d1.journal
-rw-r-----+ 1 root systemd-journal 8388608 Aug 18 05:46 system@cf1b42928e73439dbba46bbd3d87b361-0000000000002dd2-00053a498b62a4e8.journal
-rw-r-----+ 1 root systemd-journal 16777216 Aug 19 16:30 system.journal
[root@[REDACTED] td-agent]# ls -lR /var/lib/td-agent/pos/
/var/lib/td-agent/pos/:
total 4
-rw-r--r-- 1 td-agent td-agent 65 Aug 19 16:20 td-agent.pos
[root@[REDACTED] td-agent]# ls -l /var/log/td-agent/
total 20200
drwxr-xr-x 2 td-agent td-agent 479232 Aug 19 13:54 docker-containers
-rw-r----- 1 td-agent td-agent 55916 Aug 19 16:25 td-agent.log
-rw-r--r-- 1 td-agent td-agent 413280 Aug 12 03:19 td-agent.log-20160812.gz
-rw-r----- 1 td-agent td-agent 16740 Aug 13 03:28 td-agent.log-20160813.gz
-rw-r----- 1 td-agent td-agent 353 Aug 14 03:17 td-agent.log-20160814.gz
-rw-r----- 1 td-agent td-agent 330 Aug 15 03:28 td-agent.log-20160815.gz
-rw-r----- 1 td-agent td-agent 859 Aug 16 03:34 td-agent.log-20160816.gz
-rw-r----- 1 td-agent td-agent 19487686 Aug 16 19:47 td-agent.log-20160817.gz
-rw-r----- 1 td-agent td-agent 192961 Aug 19 03:46 td-agent.log-20160819
-rw-r--r-- 1 td-agent td-agent 2560 Aug 19 16:25 test.20160728.b53a7278e2176e774
[root@[REDACTED] td-agent]# cat /var/log/td-agent/test.20160728.b53a7278e2176e774
20160728T142403+0000 systemd.docker-engine {"PRIORITY":"6","UID":"0","GID":"0","SYSTEMD_SLICE":"system.slice","BOOT_ID":"9bdaf4b9512f493c99a8be69ae697588","MACHINE_ID":"ab6857580005461cab71a95470ac18e6","HOSTNAME":"[REDACTED]","SYSLOG_FACILITY":"3","CAP_EFFECTIVE":"1fffffffff","TRANSPORT":"stdout","SYSLOG_IDENTIFIER":"docker","MESSAGE":"time=\"2016-07-28T10:24:03.361026938-04:00\" level=info msg=\"New containerd process, pid: 2089\\n\"","PID":"941","COMM":"docker","EXE":"/usr/bin/docker","CMDLINE":"/usr/bin/docker daemon -H fd://","SYSTEMD_CGROUP":"/system.slice/docker.service","SYSTEMD_UNIT":"docker.service"}
20160728T142403+0000 systemd.docker-engine {"PRIORITY":"6","UID":"0","GID":"0","SYSTEMD_SLICE":"system.slice","BOOT_ID":"9bdaf4b9512f493c99a8be69ae697588","MACHINE_ID":"ab6857580005461cab71a95470ac18e6","HOSTNAME":"[REDACTED]","SYSLOG_FACILITY":"3","CAP_EFFECTIVE":"1fffffffff","TRANSPORT":"stdout","SYSLOG_IDENTIFIER":"docker","MESSAGE":"time=\"2016-07-28T10:24:03.361026938-04:00\" level=info msg=\"New containerd process, pid: 2089\\n\"","PID":"941","COMM":"docker","EXE":"/usr/bin/docker","CMDLINE":"/usr/bin/docker daemon -H fd://","SYSTEMD_CGROUP":"/system.slice/docker.service","SYSTEMD_UNIT":"docker.service"}
20160728T142403+0000 systemd.docker-engine {"PRIORITY":"6","UID":"0","GID":"0","SYSTEMD_SLICE":"system.slice","BOOT_ID":"9bdaf4b9512f493c99a8be69ae697588","MACHINE_ID":"ab6857580005461cab71a95470ac18e6","HOSTNAME":"[REDACTED]","SYSLOG_FACILITY":"3","CAP_EFFECTIVE":"1fffffffff","TRANSPORT":"stdout","SYSLOG_IDENTIFIER":"docker","MESSAGE":"time=\"2016-07-28T10:24:03.361026938-04:00\" level=info msg=\"New containerd process, pid: 2089\\n\"","PID":"941","COMM":"docker","EXE":"/usr/bin/docker","CMDLINE":"/usr/bin/docker daemon -H fd://","SYSTEMD_CGROUP":"/system.slice/docker.service","SYSTEMD_UNIT":"docker.service"}
20160728T142403+0000 systemd.docker-engine {"PRIORITY":"6","UID":"0","GID":"0","SYSTEMD_SLICE":"system.slice","BOOT_ID":"9bdaf4b9512f493c99a8be69ae697588","MACHINE_ID":"ab6857580005461cab71a95470ac18e6","HOSTNAME":"[REDACTED]","SYSLOG_FACILITY":"3","CAP_EFFECTIVE":"1fffffffff","TRANSPORT":"stdout","SYSLOG_IDENTIFIER":"docker","MESSAGE":"time=\"2016-07-28T10:24:03.361026938-04:00\" level=info msg=\"New containerd process, pid: 2089\\n\"","PID":"941","COMM":"docker","EXE":"/usr/bin/docker","CMDLINE":"/usr/bin/docker daemon -H fd://","SYSTEMD_CGROUP":"/system.slice/docker.service","SYSTEMD_UNIT":"docker.service"}
from fluent-plugin-systemd.
errm
commented on May 24, 2024
This one is hard to pin down... as far as the logs are concerned there doesn't seem to be anything going wrong in the plugin itself... it could be an underlying thing with systemd, or a particular journal...
Have you been able to reproduce the behaviour on another system... we do test the plugin against centos...
from fluent-plugin-systemd.
macdjord
commented on May 24, 2024
No - all our machines are either CentOS 7, like above, or CentOS 6, which doesn't use systemd and therefore isn't a valid target for this plugin anyway.
from fluent-plugin-systemd.
warmfusion
commented on May 24, 2024
@repeatedly suggests this may be related to treasure-data/omnibus-td-agent#87
from fluent-plugin-systemd.
errm
commented on May 24, 2024
Thanks @warmfusion...
@macdjord could you try updating td-agent and see if that solves your issue?
from fluent-plugin-systemd.
repeatedly
commented on May 24, 2024
td-agent including above fix is not released yet.
I'm now working on it but I have several building issues...
from fluent-plugin-systemd.
errm
commented on May 24, 2024
Looks like treasure-data/omnibus-td-agent#87 was closed, so this should be fixed there. If the issue persists please feel free to comment here and I will investigate.
from fluent-plugin-systemd.
Related Issues (20)
- Error: Is a directory @ io_getpartial - /var/log/journal HOT 2
- Is it possible to transfer the ownership to fluent-plugins-nursery? HOT 7
- Does support Journal Namespaces? HOT 4
- Parser type filter not working with systemd logs
- Unable to read and output journal files from default path var/log/journal HOT 1
- No logs are read on systemd 253+ potentially HOT 2
- Can't configure storage with the conf.arg parameter HOT 1
- `checking for ffi.h... *** extconf.rb failed ***` HOT 1
- How to get Gke node reboot and shutdown logs using journald Systemd service unit HOT 1
- Do not install the plugin HOT 1
- Not able to ship logs even after adding td-agent user to systemd-journal group HOT 2
- heads up! - systemd 246 journal logs produced is not readable by older versions of systemd-journald HOT 2
- support for Fedora based systems? HOT 1
- Logs getting delayed
- This plugin does not seem to work when systemd rotates the journal
- Plugin crash during graceful shutdown
- Does this plugin support zstd compression compression?
- Latest 1.0.5 version is missing some gem dependencies
- Unable to read journal files from subdirectories under default path var/log/journal
- Using systemd entry in record_modifier
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-plugin-systemd.