al42and / cuda-smi

Simple utility to show nVidia GPU memory usage wrt. CUDA device IDs.

cuda-smi

A simple utility to show nVidia GPU memory usage. Unlike nvidia-smi, it uses CUDA device IDs.

For a number of reasons nVidia uses different device enumeration in nvidia-smi monitoring utility and in their CUDA API, making it extremely frustrating to choose vacant GPU for calculations on multi-GPU machine. This utility was made to solve this problem.

Code is distributed under MIT license, except nvml.h header which is property of NVIDIA Corporation.

CUDA 7.0

With the release of CUDA 7.0, it became possible to use nvidia-smi device order in CUDA applications by setting environment variable CUDA_DEVICE_ORDER=PCI_BUS_ID. This makes this tool slightly less useful.

More information available in official docs.

Building

The code is compiled statically to simplify distribution over a large number of machines.

Simply install more-or-less recent CUDA Toolkit and run make.

Output example

aland@NX8-1:~$ cuda-smi 
Device  0 [nvidia-smi  2]:      GeForce GTX 680 (CC 3.0):     9 of  2047 MiB Used [PCIe ID: 0000:13:00.0]
Device  1 [nvidia-smi  3]:          Tesla C1060 (CC 1.3):     3 of  4095 MiB Used [PCIe ID: 0000:14:00.0]
Device  2 [nvidia-smi  1]:          Tesla C1060 (CC 1.3):   106 of  4095 MiB Used [PCIe ID: 0000:0d:00.0]
Device  3 [nvidia-smi  0]:          Tesla C2075 (CC 2.0):    13 of  6143 MiB Used [PCIe ID: 0000:0c:00.0]
Device  4 [nvidia-smi  7]:          Tesla C1060 (CC 1.3):   106 of  4095 MiB Used [PCIe ID: 0000:8e:00.0]
Device  5 [nvidia-smi  6]:          Tesla C2075 (CC 2.0):   115 of  6143 MiB Used [PCIe ID: 0000:8d:00.0]
Device  6 [nvidia-smi  5]:          Tesla C1060 (CC 1.3):   106 of  4095 MiB Used [PCIe ID: 0000:87:00.0]
Device  7 [nvidia-smi  4]:          Tesla C2075 (CC 2.0):   115 of  6143 MiB Used [PCIe ID: 0000:86:00.0]

cuda-smi's Issues

compile error, missing parameter -lnvidia-ml

i think theres a problem with your Makefile.

heres the error im getting:

g++ -I/usr/local/cuda//include -L/usr/local/cuda//lib64/ cuda-smi.cpp -lnvidia-ml -lcudart_static -lpthread -ldl -lrt -o cuda-smi
/usr/bin/ld: cannot find -lnvidia-ml
collect2: error: ld returned 1 exit status
make: *** [cuda-smi] Fehler 1

some sys specs:

cuda toolkit version 7.0 from nvidia.com resides in /usr/local/cuda
cuda management library newest installed in global scope (as of 30-5-2015)
g++ version (Ubuntu 4.8.2-19ubuntu1) 4.8.2
sys Description: Ubuntu 14.04.2 LTS
nvidia module version: nvidia-349, 349.16, 4.0.1, x86_64: installed

hope youre able to help me out.
greetings
Oliver