Exception emitting record: "\xC2" from ASCII-8BIT to UTF-8 about fluent-plugin-systemd

[errm]

potentially related to #27

[dannyk81]

Indeed seems related! any leads into what's going on?

[dannyk81]

@errm

Gentle ping :)

[errm]

Hi @dannyk81 so there seems to be two things going on here ... when the journal file is rotated logs top being ingested ... and the encoding issue in your logs.

I think these two things are unrelated but the journal file rotation issue is the root cause of the symptoms (not receiving events), the encoding thing should only cause you to lose that one event.

I haven't had the bandwidth to look at either issue yet, but may be able to over the next week at some point...

[dannyk81]

Thanks for the feedback 👍

Indeed the file rotation issue is the one that troubles me at this point.

[errm]

Re: the encoding issue it sounds like LANG is not setup correctly.

You probably want something like this in your Dockerfile:

RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8

[dannyk81]

Interesting, will give it a try!

[errm]

Humm ignore that, it seems like fluentd is specifically overriding the system encoding...

see fluent/fluentd#803 and fluent/fluentd@32790dc

[errm]

I have had a little look at this, the systemd plugin is correctly (as per the fluentd config) outputting the data with the ASCII-8BIT encoding, it seems like an output plugin that you are using is trying to convert to UTF-8, as I understand fluentd, this is a bug with that plugin as fluentd expects internally everything to be in the ASCII-8BIT encoding.

[dannyk81]

Understood, I'll have to look into that plugin internals (splunkapi), though we will probably switch to Elasticsearch soon.