Dmesg logs about fluent-plugin-systemd

[errm]

As I understand dmesg logs are copied to the journal, so some config like filters [{ "_TRANSPORT": "kernel" }] should give you similar behaviour...

Clearly you must read from the head of the journal (rather than the tail) or the cursor will already have scrolled past the kernel logs...

[bkupidura]

This is my config for systemd (no filters ...):

<source>
  @type systemd
  @label @systemd
  tag systemd.source
  path /run/log/journal
  pos_file /var/lib/fluentd/positiondb/systemd.source.pos
  <entry>
    field_map {"MESSAGE": "Payload", "_CMDLINE": "process", "_COMM": "programname", "_PID": "Pid", "_SYSTEMD_UNIT": "service", "priority": "Severity", "syslog_identifier": "ident" }
    field_map_strict true
    fields_strip_underscores true
    fields_lowercase true
  </entry>
</source>

When new messages lands in dmesg (ex. OOM), they are not collected.

[errm]

And you are seeing the message with journalctl -k ?

[bkupidura]

Yep, correct. I disabled everything except source + final match and i see those events. Give me some time and i will debug what is wrong with my config...

[errm]

Humm, I am not sure off hand...

Could you share a little more info about your setup, specifically OS / systemd versions...

Some thoughts that might help with debugging this:

• Try adding filters [{ "_TRANSPORT": "kernel" }] anyway, perhaps libsystemd has an implicit default to filter out these messages...
• Is this a permissions issue? Try running fluentd as root to see if that makes any difference (just to debug)
• this plugin uses https://github.com/ledbettj/systemd-journal under the hood to read from the journal, if dmseg stuff really is copied to the journal it should be readable like any other entry...
• perhaps your systemd is not correctly copying the entries from the kernel ring buffer to the journal (I don't know how that happens... but I guess it must be a thing)

[bkupidura]

So yep, this is issue with my env. My config use rewrite_tag_filter based on service field (which is created from _SYSTEMD_UNIT.

<match systemd.source>
  @type rewrite_tag_filter
  <rule>
    key service
    pattern ^(.*)\.(.*)$
    tag __TAG__.$1
  </rule>
</match>

Normal messages have service fields:

2018-01-08T09:52:03+00:00       systemd.source.ssh      {"Payload":"Accepted publickey for root from 172.16.10.90 port 50048 ssh2: RSA SHA256:DCSSfinZ+IFdLFh2SuvfdD3u5pcm8dG8zbVW8SQ1vyI","process":"sshd: root [priv]   ","programname":"sshd","Pid":"7827","service":"ssh.service","Severity":"6","ident":"sshd","severity_label":"INFO", "Timestamp":"2018-01-08T09:52:03+0000"}

But kernel messages miss service fields:

2018-01-08T10:11:19+00:00       systemd.source.kernel   {"Payload":"Out of memory: Kill process 10806 (a.out) score 650 or sacrifice child","Severity":"3","ident":"kernel","severity_label":"ERROR"}

So final result is that my kernel messages are dropped. Solution for that is to use also ident:

<match systemd.source>
  @type rewrite_tag_filter
  <rule>
    key service
    pattern ^(.*)\.(.*)$
    tag __TAG__.$1
  </rule>
  <rule>
    key ident
    pattern ^(.*)$
    tag __TAG__.$1
  </rule>
</match>

We can close that issue, Thanks!

[errm]

Yay, thanks for letting me know... I am glad that this does indeed work as I would expect