[Bug]: nvidia_smi collector didn't start until running the debug command once about netdata

[ilyam8]

Hi, @gregjhogan. I can't reproduce the issue.

Running go.d.plugin in debug mode has nothing to do with working/not working when running as a systemd service.

You are right:

• the "nvidia_smi" module is disabled by default and needs to be manually enabled in "go.d.conf".
• you need to restart Netdata after modifying changes to "go.d.conf".

These are your go.d/nvidia_smi related logs:

Jun 07 20:32:44 gpu-server-11 netdata[45841]: level=info msg="check success" plugin=go.d collector=nvidia_smi job=nvidia_smi
Jun 07 20:32:44 gpu-server-11 netdata[45841]: level=info msg="started, data collection interval 10s" plugin=go.d collector=nvidia_smi job=nvidia_smi
Jun 07 21:30:01 gpu-server-11 netdata[45841]: level=info msg=stopped plugin=go.d collector=nvidia_smi job=nvidia_smi

Jun 12 18:19:17 gpu-server-11 netdata[403331]: level=info msg="check success" plugin=go.d collector=nvidia_smi job=nvidia_smi
Jun 12 18:19:17 gpu-server-11 netdata[403331]: level=info msg="started, data collection interval 10s" plugin=go.d collector=nvidia_smi job=nvidia_smi
Jun 12 18:55:41 gpu-server-11 netdata[403331]: level=info msg=stopped plugin=go.d collector=nvidia_smi job=nvidia_smi

Jun 12 18:56:08 gpu-server-11 netdata[435423]: level=info msg="check success" plugin=go.d collector=nvidia_smi job=nvidia_smi
Jun 12 18:56:08 gpu-server-11 netdata[435423]: level=info msg="started, data collection interval 10s" plugin=go.d collector=nvidia_smi job=nvidia_smi

As you see the collector was stopped on Jun 7 an hour after the start. I don't know why, you need to check other logs at the time.

[gregjhogan]

Yeah it was a weird one, as I haven't had this problem in the past. As you can see from the UI screen shots, even if it was running it wasn't collecting any data. If you want any additional logs, just let me know what to send and where to send them.

[Thendon]

Had the same issue, restart the docker container ~10 times did not help. Only solution was running the debug command for some time and restart the container again as described above.

The logs always contained the lines, even when nvidia_smi didn't show up:
netdata | time=2024-06-19T21:54:11.614Z level=info msg="check success" plugin=go.d collector=nvidia_smi job=nvidia_smi netdata | time=2024-06-19T21:54:11.614Z level=info msg="started, data collection interval 10s" plugin=go.d collector=nvidia_smi job=nvidia_smi

[gregjhogan]

@ilyam8 I just had it happen again and this time it was on a server that was previously working.

• UI last shows data for nvidia smi on June 19, 2024 (but noticed today)
• Today I try restarting the netdata service several times but still no data showing up in the UI for nvidia smi
• Run the command ./go.d.plugin -d -m nvidia_smi and everything looks good
• Restart the netdata service again and I see new data coming in now for nvidia smi

That was everything I did to get it working again, so no config changed between it not working and then working again.