Corrupted journal can still cause crashes about fluent-plugin-systemd

[JoshuaOliphant]

This issue is happening for me too. In fluentd logs:

2019-05-30 01:52:12.729014141 +0000 fluent.error: {"title":"in_systemd_emit_worker","error":"#<Systemd::JournalError: Bad message>","message":"[journald] Unexpected error raised. Stopping the timer. title=:in_systemd_emit_worker error_class=Systemd::JournalError error=\"Bad message\""}

and

2019-05-30 01:52:12 +0000 [error]: #0 [journald] Unexpected error raised. Stopping the timer. title=:in_systemd_emit_worker error_class=Systemd::JournalError error="Bad message

Received these 2 logs a single time. The logs on this particular kubernetes pod stopped flowing. I deleted the pod, it came back up, and then the logs started flowing again.

I ran journalctl --verify from the node where the pod was running, but also nodes there the pod was NOT running, and saw the same FAIL messages. They came and went in both cases, so I don't think that was important.

[mariusgrigoriu]

Here's the full log related to this

2019-05-31 16:52:15 +0000 [warn]: #0 [journald] Error reading from Journal: Systemd::JournalError: Bad message
2019-05-31 16:52:15 +0000 [error]: #0 [journald] Unexpected error raised. Stopping the timer. title=:in_systemd_emit_worker error_class=Systemd::JournalError error="Bad message"
  2019-05-31 16:52:15 +0000 [error]: #0 /usr/local/bundle/gems/systemd-journal-1.3.3/lib/systemd/journal/navigable.rb:44:in `move_next'
  2019-05-31 16:52:15 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-systemd-1.0.2/lib/fluent/plugin/in_systemd.rb:131:in `watch'
  2019-05-31 16:52:15 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-systemd-1.0.2/lib/fluent/plugin/in_systemd.rb:109:in `run'
  2019-05-31 16:52:15 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.4.0/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
  2019-05-31 16:52:15 +0000 [error]: #0 /usr/local/bundle/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2019-05-31 16:52:15 +0000 [error]: #0 /usr/local/bundle/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2019-05-31 16:52:15 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.4.0/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2019-05-31 16:52:15 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.4.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-05-31 16:52:15 +0000 [error]: #0 [journald] Timer detached. title=:in_systemd_emit_worker
2019-05-31 16:52:15.913211850 +0000 fluent.warn: {"message":"[journald] Error reading from Journal: Systemd::JournalError: Bad message"}
2019-05-31 16:52:15.916827918 +0000 fluent.error: {"title":"in_systemd_emit_worker","error":"#<Systemd::JournalError: Bad message>","message":"[journald] Unexpected error raised. Stopping the timer. title=:in_systemd_emit_worker error_class=Systemd::JournalError error=\"Bad message\""}
2019-05-31 16:52:15.917048234 +0000 fluent.error: {"title":"in_systemd_emit_worker","message":"[journald] Timer detached. title=:in_systemd_emit_worker"}

[errm]

Thanks to the contribution of @mariusgrigoriu this should now be fixed, a new version has been released https://github.com/fluent-plugin-systemd/fluent-plugin-systemd/releases/tag/v1.0.3