Grok exporter show changes only after restart about grok_exporter

[fstab]

If I understand you correctly, you want that grok_exporter picks up changes of the configuration file without restart. Unfortunately, this is currently not supported. grok_exporter reads the config file once on startup. The configuration cannot be changed during runtime.

The poll_interval_seconds defines how often the logfile is checked for new log lines. It does not specify how often the configuration file is reloaded. In most cases, you will not need this setting, because grok_exporter hooks into the operating system's file system notifications and gets notified about new log lines without polling, see CONFIG.md.

However, it should not be a problem to restart grok_exporter. If you are afraid because counters are reset to 0 after restart: This is ok, because the Prometheus Server covers this. For example, if you query an increase rate of a counter in Prometheus (for example rate(http_requests_total[5m])), you will get the correct result even if the counter is reset to 0 within the 5m time interval. This is meant by "breaks in monotonicity ... are automatically adjusted for" in the Prometheus documentation.

[Priyotosh008]

Than you for your reply.

What i meant is NOT grok_exporter picking up the configuration file without restart.

My problem is:
I have configured grok to match patters from the log file.

Config.yml:

---

global:
    config_version: 2
input:
    type: file
    path: /ZAMBAS/logs/Healthcheck/EMS/Health_Status.log
    readall: true
    poll_interval_seconds: 5

grok:
    patterns_dir: ./patterns

metrics:
    - type: gauge
      name: EMSINT
      help: Counter metric example with labels.
      match: '%{USERNAME:Machine}\s%{WORD:Instance}\sstate\s:\s%{USERNAME:State}'
      value: '{{.State}}'
      cumulative: false
      labels:
          Machine: '{{.Machine}}'
          Instance: '{{.Instance}}'

server:
    port: 9744

LogFile Content:

---

AWEMUCLHI601 SST state : 2
AWEMUCLHI601 SSL state : 2
AWEMUCLHI602 SST state : 1
AWEMUCLHI602 SSL state : 2
AWEMUCLHI603 SST state : 1
AWEMUCLHI603 SSL state : 2
AWEMUCLHI604 SST state : 2
AWEMUCLHI604 SSL state : 2
AWEMUCLHI601 ASyncOUT state : 2
AWEMUCLHI602 ASyncOUT state : 0
AWEMUCLHI601 Trans state : 2
AWEMUCLHI602 Trans state : 0
AWEMUCLHI603 AUAAsyncEXT01 state : 2

PROBLEM STATEMENT:

---

Now my intention is NOT to count the number of matching lines. I want to get the value (2,1,0).
So whenever i restart the grok instance it updates correctly. But while grok is already running and the values gets changed in the log file, this doesn't reflects/updates.

[fstab]

Sorry, I got you wrong. Getting updates from log lines should work. If you have a poll_interval_seconds of 5, you should see the updated value after max. 5 seconds. Can you tell me which operating system and file system you use? Then I will try to reproduce the error in a virtual machine. I tried it on macOS, end everything worked fine, so it's probably something OS specific.

[Priyotosh008]

Operating System: Linux
[prometheus@AWEMUCLHI601 ~]$ uname -a
Linux 2.6.32-696.16.1.el6.x86_64 #1 SMP Sun Oct 8 09:45:56 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

File System:
Log file Location - type nfs
Grok config location - type ext4

[actional@AWEMUCLHI601 ~]$ mount | column -t|grep -i "logs"
awemuclhinas:/vol/vol_int_ENF/int/ZAMBAS/logs on /ZAMBAS/logs type nfs

[fstab]

I am having trouble to reproduce the error. I set up two Ubuntu VMs on Digitalocean, installed the nfs server on one and the nfs client on the other. Then I ran grok_exporter on the client with your config from above and started adding a few log lines with different data on the server:

echo 'AWEMUCLHI603 AUAAsyncEXT01 state : 42' >> /logs/Health_Status.log

After max. 5 seconds the updated number 42 appears on grok_exporter's /metrics endpoint as expected: EMSINT{Instance="AUAAsyncEXT01",Machine="AWEMUCLHI603"} 42

Maybe it's something related to the nfs version. When I run mount on the client I see type nfs4, but you say above the mount output on your system shows nfs without the 4. Which nfs version are you using?

[Priyotosh008]

We are using nfs version 3
nfsstat -v
Client packet stats:
packets udp tcp tcpconn
0 0 0 0

Client rpc stats:
calls retrans authrefrsh
53802840 13 53803069

Client nfs v3:
null getattr setattr lookup access readlink
0 0% 18067449 33% 244278 0% 1242923 2% 4725464 8% 0 0%
read write create mkdir symlink mknod
15903287 29% 12000761 22% 144108 0% 9654 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
156011 0% 9728 0% 48579 0% 0 0% 39070 0% 763513 1%
fsstat fsinfo pathconf commit
448041 0% 12 0% 6 0% 0 0%

[Priyotosh008]

Also, for the first time when we start the grok instance it picks the values from log file correctly. But while the grok is already running and then after some time the values gets changed, it doesn't reflects without a restart of the respective grok instance.

[fstab]

I tried NFSv3, but still it works fine. Could you do an experiment and run watch tail /ZAMBAS/logs/Healthcheck/EMS/Health_Status.log on the same machine where grok_exporter is running? The watch command will force the log file to be read every 2 seconds. You will see in the watch output if it works. I am curious whether grok_exporter picks up the changes while watch is running or not.

[fstab]

Closing the issue because I wasn't able to reproduce it. If you are still running into this, please feel free to re-open and tell me if other programs (like tail -f) get updates for the files or not. It could be a general filesystem problem not related to grok_exporter, but in that case tail -f should not work.