grok - log rotation issue about grok_exporter

[senattcsgit]

I upgraded the grok and so far I didnt see the error I used to see with previous release. Heading in right direction, but I would like to monitor for few more days to close this issue.

[fstab]

This error occurs if the grok_exporter process does not have the permission to open the log file. Could you double-check if the user running grok_exporter can read the file?

[senattcsgit]

file permissions are okay because grok exporter comes up successfully and parses log files but when log file gets rotated-thats when its get killed. if permissions were problem, i would have seen issue when Im bringing up grok initially.

Log files have 755 permissions and grok has rwx for these files.

[fstab]

How is your logrotate configured and which file system are you using?

[senattcsgit]

We are using rsyslog.d and here is the configuration 👍
[xxxx@ppppppp002c:/etc/rsyslog.d]> cat haproxy.conf

Saving application haproxy logs haproxy.log

$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1
local0.* /var/prod/logs/yyyy/haproxy/tugs_hap.log
local1.* /var/prod/logs/yyyy/haproxy/http_hap.log
local2.* /var/prod/logs/yyyy/haproxy/securehttp_hap.log
local4.* /var/prod/logs/yyyy/haproxy/rest_hap.log
local5.* /var/prod/logs/yyyy/haproxy/ssl_hap.log
local6.* /var/prod/logs/yyyy/haproxy/caas_hap.log
[xxxx@ppppppp002c:/etc/rsyslog.d]>

[senattcsgit]

and the filesystem type is xfs

/dev/mapper/vg01-lvvarprodlogsxxxx /var/prod/logs/xxxx xfs defaults 1 2

[fstab]

Maybe the error is related to how the rotated files like http_hap.log-20190529.gz are created, and how the logfile http_hap.log is re-created after rotation. Is rsyslog doing this in your case, or is logrotate running in addition to rsyslog to rotate the log files?

I am asking because rsyslog seems to support custom log rotation scripts (like here), and logrotate has a lot of configuration options how log files are rotated and re-created.

[senattcsgit]

Any best practices you can advice on the log rotation like moving the old file or copying? How is community handling these scenarios, I'm sure no one is maintaining one log file, which might fill up the disk space...Thoughts? In the meantime, I will get answers on rsyslog as it is handled by UNIX SA's in my company.

[senattcsgit]

Looks like it's both rsyslog and logrotate

and here is the logrotate configuration:

[xxxx@ppppppp002c:/etc/logrotate.d]> cat haproxy
/var/prod/logs/xxxx/haproxy/tugs_hap.log
/var/prod/logs/xxxx/haproxy/http_hap.log
/var/prod/logs/xxxx/haproxy/securehttp_hap.log
/var/prod/logs/xxxx/haproxy/rest_hap.log
/var/prod/logs/xxxx/haproxy/ssl_hap.log
/var/prod/logs/xxxx/haproxy/caas_hap.log
{
missingok
notifempty
sharedscripts
size 500M
rotate 4
daily
compress
postrotate
service rsyslog restart >/dev/null 2>&1 || true
endscript
create 0755 xxxx yyyy
}
[xxxx@ppppppp002c:/etc/logrotate.d]>

[fstab]

This should work. Could you update to grok_exporter v0.2.7 and report if the error persists, and which error message you get with the current release?

[senattcsgit]

current error message : error reading log lines: failed to watch /var/prod/logs/xxxx/haproxy/http_hap.log: open /var/prod/logs/xxxx/haproxy/http_hap.log: permission denied

I will work on upgrade and report tomorrow . Thanks for your help.

[senattcsgit]

Even with new grok version- I'm still seeing the issue.
grok version: grok_exporter version: 0.2.7 (build date: 2019-04-08, branch: master, revision: e2ba841, go version: go1.12.2, platform: linux-amd64)

error reading log lines: /var/prod/logs/xxxx/haproxy/http_hap.log: failed to open file: open /var/prod/logs/xxxx/haproxy/http_hap.log: permission denied

Is this happening when log rotation is in progress and grok is trying to open file? anyway to ignore

[fstab]

I checked where the error message is created in the grok_exporter source code: I assume logrotate ran shortly before the error and moved the original http_hap.log away. Then a new http_hap.log was created. grok_exporter detects the new file, tries to open it, and gets a permission denied error from the operating system. That causes the error.

As you say grok_exporter can successfully read the file when you restart it, the permission denied error seems to be a temporary problem that occurs only shortly after the new file is created.

Could you check if you have ACLs configured for your file system? Maybe ACLs are causing a short delay until correct permissions are applied.

[senattcsgit]

Closing the issue as we found a work around by introducing a cronjob and more recently by moving to kubernetes.