sicara / gpumonitor

TF 2.x and PyTorch Lightning Callbacks for GPU monitoring

gpumonitor

gpumonitor gives you stats about GPU usage during execution of your scripts and trainings, as TensorFlow or Pytorch Lightning callbacks.

Installation

Installation can be done directly from this repository:

pip install gpumonitor

Getting started

Option 1: In your scripts

monitor = gpumonitor.GPUStatMonitor(delay=1)

# Your instructions here
# [...]

monitor.stop()
monitor.display_average_stats_per_gpu()

It keeps track of the average of GPU statistics. To reset the average and start from fresh, you can also reset the monitor:

monitor = gpumonitor.GPUStatMonitor(delay=1)

# Your instructions here
# [...]

monitor.display_average_stats_per_gpu()
monitor.reset()

# Some other instructions
# [...]

monitor.display_average_stats_per_gpu()

Option 2: Callbacks

Add the following callback to your training loop:

For TensorFlow,

from gpumonitor.callbacks.tf import TFGpuMonitorCallback

model.fit(x, y, callbacks=[TFGpuMonitorCallback(delay=0.5)])

For PyTorch Lightning,

from gpumonitor.callbacks.lightning import PyTorchGpuMonitorCallback

trainer = pl.Trainer(callbacks=[PyTorchGpuMonitorCallback(delay=0.5)])
trainer.fit(model)

Display Format

You can customize the display format according to the gpustat options. For example, display of watts consumption, fan speed are available. To know which options you can change, refer to:

• TensorFlow callback example
• gpustat print_to() docstring

Sources

• Built on top of GPUStat
• Separate thread loop coming from gputil

gpumonitor's Issues

Can't import GPUStatMonitor

Atm, running the following lines turns into an error:

import gpumonitor

gpu_monitor = gpumonitor.monitor.GPUStatMonitor

A fix is to do:

from gpumonitor import monitor

gpu_monitor = monitor.GPUStatMonitor

But we need to improve the init handling so users can access objects more efficiently

Will be updated to adapt to new PL?

Thanks for contributing such a good tool!

Add MIT licence file to the library

Print options

Users could benefit moving from the __repr__ method from GPUStatCollection to print_to

Provide Default delay value

Providing a reasonable default sampling delay, would simplify usage and avoid user error due to misunderstanding the argument. A reasonable value should be good enough for the vast majority of use cases.