wilicc / gpu-burn Goto Github PK

View Code? Open in Web Editor NEW

1.2K 18.0 273.0 72 KB

Multi-GPU CUDA stress test

License: BSD 2-Clause "Simplified" License

Makefile 3.97% Cuda 7.24% C++ 85.28% Dockerfile 0.92% Roff 2.60%

gpu-burn's Introduction

gpu-burn

Multi-GPU CUDA stress test http://wili.cc/blog/gpu-burn.html

Easy docker build and run

git clone https://github.com/wilicc/gpu-burn
cd gpu-burn
docker build -t gpu_burn .
docker run --rm --gpus all gpu_burn

Binary packages

https://repology.org/project/gpu-burn/versions

Building

To build GPU Burn:

make

To remove artifacts built by GPU Burn:

make clean

GPU Burn builds with a default Compute Capability of 5.0. To override this with a different value:

make COMPUTE=<compute capability value>

CFLAGS can be added when invoking make to add to the default list of compiler flags:

make CFLAGS=-Wall

LDFLAGS can be added when invoking make to add to the default list of linker flags:

make LDFLAGS=-lmylib

NVCCFLAGS can be added when invoking make to add to the default list of nvcc flags:

make NVCCFLAGS=-ccbin <path to host compiler>

CUDAPATH can be added to point to a non standard install or specific version of the cuda toolkit (default is /usr/local/cuda):

make CUDAPATH=/usr/local/cuda-<version>

CCPATH can be specified to point to a specific gcc (default is /usr/bin):

make CCPATH=/usr/local/bin

CUDA_VERSION and IMAGE_DISTRO can be used to override the base images used when building the Docker image target, while IMAGE_NAME can be set to change the resulting image tag:

make IMAGE_NAME=myregistry.private.com/gpu-burn CUDA_VERSION=12.0.1 IMAGE_DISTRO=ubuntu22.04 image

Usage

GPU Burn
Usage: gpu_burn [OPTIONS] [TIME]

-m X   Use X MB of memory
-m N%  Use N% of the available GPU memory
-d     Use doubles
-tc    Try to use Tensor cores (if available)
-l     List all GPUs in the system
-i N   Execute only on GPU N
-h     Show this help message

Example:
gpu_burn -d 3600

gpu-burn's People

Contributors

Stargazers

Watchers

Forkers

mincore duanmuweiyu rixon elementai nissan fmoctezuma themistymay kenanpelit naviocean madisongh zvonkok freeyssu davidstack kkoscev liu4lin hurdad nanoant earthmine noxouille c5f3 t-kalinowski eulerjiang dward4 nargetdev vanandelinstitute hoangtnm jirakj flyaudio yotabits ulandz willfurnass fladrif spaulaus chenk008 anonymouse64 dfvneto radical-cybertools howarddickens100 hubearth yohandevelter liuha jljames mjnovice umass-compsci-630 skycatch bladernr cnr-isc-ttt m4tts genesiscloud boyle saintedsheep mrkolen tcinbis harsv nextguido kindlychung lzufalcon felker enijkamp jiaoyongqi adujardin igormp hintron nicksan2c igarfieldi rncm-prism dennistang742 melanj remalcodex jolibrain siddharth-maddali ygene2 photoneo rochaporto chuanli11 anseeto stylistbw cloveryww jeremybobbin jcsma-utils lambdasupport matzmz heero-yuy admin-ipfs wilshire461 vigileagle litai686 eengstrom f0ucan rjc brandery cloudmercato liam1098 cheeeee golemherry zeta1999 shaul-yermol byandrey gzoom13 boweryfarming

gpu-burn's Issues

Install gpu-burn: Couldn't init a GPU test

I am trying to install gpu-burn to /usr/bin, but no mattter what I try I get this error since it needs to be ran from the gpu-burn folder ONLY:

Couldn't init a GPU test: Error in "couldn't find file "compare.ptx" from working directory": CUDA_ERROR_NOT_FOUND

I tried symlinks, adding to the PATH. Is there a way I can have it pull from the folder it was executed from or have a shared location of some sort?

gpu_burn-drv.cpp:50:10: fatal error: cuda.h: No such file or directory

[bin]$ git clone [email protected]:wilicc/gpu-burn.git      
Cloning into 'gpu-burn'...
remote: Enumerating objects: 56, done.
remote: Counting objects: 100% (56/56), done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 129 (delta 30), reused 44 (delta 22), pack-reused 73
Receiving objects: 100% (129/129), 34.94 KiB | 420.00 KiB/s, done.
Resolving deltas: 100% (69/69), done.
[bin]$ cd gpu-burn/
[gpu-burn]$ make
g++ -O3 -Wno-unused-result -I/usr/local/cuda/include -c gpu_burn-drv.cpp
gpu_burn-drv.cpp:50:10: fatal error: cuda.h: No such file or directory                 
   50 | #include <cuda.h>
      |          ^~~~~~~~
compilation terminated.
make: *** [Makefile:32: gpu_burn-drv.o] Error 1

Allow infinite runtime by parameter 0

Enhancement request: If time set to 0 then run infinitely.

RTX 4090 maximum power

The RTX 4090 has a maximum power usage of 450w but gpu-burn only explored around 380w in my work-station. How can we make it fully loaded 450w/450w.

Errors on Turing GPUs

With GTX 2080 Turing graphics, the program errors with the "DIED!" message.

Is there a place where I can lookup the typical gpu_burn results for various GPUs? For example, I am seeing what I consider to be a super low Gflops/s for K80 on Azure, but I don't know what it's supposed to be.

$ ./gpu_burn 
Run length not specified in the command line.  Burning for 10 secs
GPU 0: Tesla K80 (UUID: GPU-b57841d9-9b07-f573-b01c-c2e16694f1ae)
Initialized device 0 with 11441 MB of memory (11351 MB available, using 10216 MB of it), using FLOATS
50.0%  proc'd: 636 (2085 Gflop/s)   errors: 0   temps: 64 C 
	Summary at:   Fri Aug  9 22:17:04 UTC 2019

100.0%  proc'd: 636 (2085 Gflop/s)   errors: 0   temps: 64 C 
	Summary at:   Fri Aug  9 22:17:09 UTC 2019

100.0%  proc'd: 1908 (2273 Gflop/s)   errors: 0   temps: 66 C

It would be cool, if gpu_burn asks the user to publish the results at the end of the test, in exchange of looking up the test results from other users with similar hardware so that the score can be compared.

Or you could simply setup a wiki, or host a result lists and allow people to send PR with results.

intialized device 过程

[root@localhost gpu-burn-master]# ./gpu_burn 1000
GPU 0: NVIDIA A2 (UUID: GPU-da20406d-5c91-2506-a377-c919f8f6dea1)
Initialized device 0 with 14830 MB of memory (14437 MB available, using 12993 MB of it), using FLOATS

启动gpuburn时会有一段时间处于Initialized 这个状态，这个是gpuburn的程序启动嘛？对卡的状态是否有影响？我GPU卡是否处于活跃状态？

望回复！谢谢！

Failure during compute: Error in "SGEMM": CUBLAS_STATUS_EXECUTION_FAILED

does not run:

./gpu_burn 10
GPU 0: GeForce GTX 1650 (UUID: GPU-0fccd51a-9bf1-17e7-812c-c1dcf2ca2c42)
Initialized device 0 with 3914 MB of memory (3165 MB available, using 2849 MB of it), using FLOATS
Failure during compute: Error in "SGEMM": CUBLAS_STATUS_EXECUTION_FAILED
10.0% proc'd: -1 (0 Gflop/s) errors: -1 (DIED!) temps: 46 C

No clients are alive! Aborting

using GeForce GTX 1650, Ubuntu 18.04.3 LTS, NVIDIA Driver Version: 430.50, CUDA Version: 10.1

other CUDA code runs fine on my setup!

Couldn't init a GPU test: Error in "init": CUBLAS_STATUS_NOT_INITIALIZED

I'm getting this error when running the burn test.

Standard format output file

Hello,

First of all, many thanks for this app, it's great!
But, I need to implement an option to get a formatted output.

I would like start by JSON, with a -j <filename.json> option.

Maybe something like that:

{
  "devices": {
    "<DEVICE>": {
      "model": "<GPU_MODEL>",
      "UUID": "<UUID>",
      "<GLFOPS>": [<ARRAY_OF_GLOPS_VALUES>],
      "<ERROS>": [<ARRAY_OF_ERROS_VALUES>],
      "<TEMPS>": [<ARRAY_OF_TEMPS_VALUES>]
     }
  },
  "elapsed": [<ARRAY_OF_SUMMARY_TIMES>]
}

My example with 2 GPUs:

{
  "devices": {
    "GPU 0": {
      "model": "A100-SXM-80GB",
      "UUID": "GPU-3bb11c3f-033a",
      "Gflop/s": ["17668", "17699", "17705", "17704"],
      "errors": ["0", "0", "0", "0"],
      "temps (C)": ["56", "59", "60", "60"]
    },
    "GPU 1": {
      "model": "A100-SXM-80GB",
      "UUID": "GPU-c1aa22c3a-032b",
      "Gflop/s": ["17668", "17699", "17705", "17704"],
      "errors": ["0", "0", "0", "0"],
      "temps(C)": ["56", "59", "60", "60"]
    }
  }
  "elapsed(s)": ["10", "20", "30", "40"]
}

It gonna be easier to analyze and make statistics, with python or jq.

Before starting, what do you think about?

Best,
Adrien

gpu support

hi,will it work on amd gpu?

[Makefile:10: drv] Error 127

I'm on fedora 31

git clone https://github.com/wilicc/gpu-burn
Cloning into 'gpu-burn'...
remote: Enumerating objects: 19, done.
remote: Counting objects: 100% (19/19), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 66 (delta 9), reused 11 (delta 4), pack-reused 47
Unpacking objects: 100% (66/66), 25.71 KiB | 110.00 KiB/s, done.
[nikolay@localhost Downloads]$ cd gpu-burn
[nikolay@localhost gpu-burn]$ make
bash: make: command not found...
Install package 'make' to provide command 'make'? [N/y] y


 * Waiting in queue... 
 * Loading list of packages.... 
The following packages have to be installed:
 guile22-2.2.6-2.fc31.x86_64	A GNU implementation of Scheme for application extensibility
 make-1:4.2.1-15.fc31.x86_64	A GNU tool which simplifies the build process for users
Proceed with changes? [N/y] y


 * Waiting in queue... 
 * Waiting for authentication... 
 * Waiting in queue... 
 * Downloading packages... 
 * Requesting data... 
 * Testing changes... 
 * Installing packages... 
PATH=/home/nikolay/.local/bin:/home/nikolay/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/nikolay/.composer/vendor/bin:/var/lib/snapd/snap/bin:.:/usr/bin:/home/nikolay/.local/bin:/home/nikolay/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/nikolay/.composer/vendor/bin:/var/lib/snapd/snap/bin /usr/local/cuda/bin/nvcc -I/usr/local/cuda/include -arch=compute_30 -ptx compare.cu -o compare.ptx
/bin/sh: /usr/local/cuda/bin/nvcc: No such file or directory
make: *** [Makefile:10: drv] Error 127

Core dump after trying to test the latest V100 Tesla cards

This fails on a system with 16 GPGPUs (Tesla V100 SXM3s)

$ /opt/gpu-burn$ ./gpu_burn
Run length not specified in the command line. Burning for 10 secs
GPU 0: Tesla V100-SXM3-32GB (UUID: GPU-0baad8c1-90f1-a0cd-bbd6-5bfdaeb6a747)
GPU 1: Tesla V100-SXM3-32GB (UUID: GPU-63cb6f99-dce9-171f-1e51-b4f82ac54a2a)
GPU 2: Tesla V100-SXM3-32GB (UUID: GPU-fd6dee30-31a6-e0d9-5bea-2f10f4f4adf9)
GPU 3: Tesla V100-SXM3-32GB (UUID: GPU-40b208b4-c598-d125-1f4d-428127b62a00)
GPU 4: Tesla V100-SXM3-32GB (UUID: GPU-0c2b7ba6-ca06-dccb-8fa2-a97e4cd3ba9b)
GPU 5: Tesla V100-SXM3-32GB (UUID: GPU-3b3a305f-3e5b-c5cf-031b-d96dfc9f19cc)
GPU 6: Tesla V100-SXM3-32GB (UUID: GPU-178881f5-c6b0-4de3-309a-0eac0a6153bd)
GPU 7: Tesla V100-SXM3-32GB (UUID: GPU-e7e5c623-6b17-9598-056f-cad0ff599286)
GPU 8: Tesla V100-SXM3-32GB (UUID: GPU-2242ae02-9e04-7fcf-f287-b4d17e555d57)
GPU 9: Tesla V100-SXM3-32GB (UUID: GPU-ad843060-d718-47a9-b1d6-117a7870123a)
GPU 10: Tesla V100-SXM3-32GB (UUID: GPU-d36440ed-54fc-c4d0-5288-93ed6375795c)
GPU 11: Tesla V100-SXM3-32GB (UUID: GPU-894366f9-451b-d088-f2cb-ee4037dc19da)
GPU 12: Tesla V100-SXM3-32GB (UUID: GPU-c578fe7a-d3f5-09fd-a847-66d5f6ad3b5c)
GPU 13: Tesla V100-SXM3-32GB (UUID: GPU-e4a6f9ca-77dd-3895-d0ad-29430cc610b3)
GPU 14: Tesla V100-SXM3-32GB (UUID: GPU-9ebe7746-ac18-ba65-6117-319681c32f7c)
GPU 15: Tesla V100-SXM3-32GB (UUID: GPU-d7804840-3d39-7b1b-2dc0-f301be7f2664)
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
*** buffer overflow detected ***: ./gpu_burn terminated
Aborted (core dumped)

At this point, it seems to just keep spawning instances of gpu_burn:
$ ps axf |grep gpu_burn |wc -l
20544
Here's 20,544 instances that I think are all in the Dead state:
24750 ? D 0:00 ./gpu_burn

ce:00.0 Bridge: NVIDIA Corporation Device 1ac2 (rev a1)
Flags: bus master, fast devsel, latency 0, IRQ 928, NUMA node 1
Memory at e1000000 (32-bit, non-prefetchable) [size=16M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024
Kernel driver in use: nvidia-nvswitch

These are all connected via NVLink.

Running the latest cuda stack from developer.nvidia.com on Ubuntu 18.04 with the 4.15 kernel.

gpu_burn was pulled from git and compiled just before trying the test.

Wrong result when error happens after the last progress report

Hello!

First of all I'd like to thank every contributor for their work, gpu burn is a very useful tool for us.

I think we found a bug in the code that prints out the final results. We used the tool to test a machine with 3 GPUs for 90 minutes. I'm attaching the stdout of gpu burn: 20221207_190200_gpu_burn_stdout.zip

From the output you can see that around the end of the test (at 92.2%) GPU 1 started to produce some errors:

92.2%  proc'd: 2710955 (9344 Gflop/s) - 2744965 (9469 Gflop/s) - 2710060 (9341 Gflop/s)   errors: 0 - 0 - 0   temps: 70 C - 70 C - 81 C 
92.2%  proc'd: 2710955 (9344 Gflop/s) - 2744965 (9469 Gflop/s) - 2710060 (9341 Gflop/s)   errors: 0 - 0 - 0   temps: 70 C - 70 C - 81 C 
92.2%  proc'd: 2710955 (9344 Gflop/s) - 2744965 (9469 Gflop/s) - 2710060 (9341 Gflop/s)   errors: 0 - 0 - 0   temps: 70 C - 70 C - 81 C 
92.2%  proc'd: 2710955 (9344 Gflop/s) - 2744965 (9469 Gflop/s) - 2710060 (9341 Gflop/s)   errors: 0 - 0 - 0   temps: 70 C - 70 C - 81 C 
92.2%  proc'd: 2711850 (9347 Gflop/s) - 2744965 (9469 Gflop/s) - 2710060 (9341 Gflop/s)   errors: 1  (WARNING!)- 0 - 0   temps: 70 C - 70 C - 81 C 
92.2%  proc'd: 2711850 (9347 Gflop/s) - 2744965 (9469 Gflop/s) - 2710955 (9347 Gflop/s)   errors: 1  (WARNING!)- 0 - 0   temps: 70 C - 70 C - 81 C 
92.3%  proc'd: 2711850 (9347 Gflop/s) - 2745860 (9459 Gflop/s) - 2710955 (9347 Gflop/s)   errors: 1  (WARNING!)- 0 - 0   temps: 70 C - 70 C - 81 C 
92.3%  proc'd: 2712745 (9341 Gflop/s) - 2745860 (9459 Gflop/s) - 2710955 (9347 Gflop/s)   errors: 1  (WARNING!)- 0 - 0   temps: 70 C - 70 C - 81 C 
92.3%  proc'd: 2712745 (9341 Gflop/s) - 2745860 (9459 Gflop/s) - 2711850 (9343 Gflop/s)   errors: 1  (WARNING!)- 0 - 0   temps: 70 C - 70 C - 81 C

However the final report says:

Tested 3 GPUs:
	GPU 0: OK
	GPU 1: OK
	GPU 2: OK

The problem is that the clientFaulty.at(i) = true; statement is executed only when the intermediate summary reports are printed.

gpu-burn/gpu_burn-drv.cpp

Lines 619 to 632 in 454d1bf

 if (nextReport < elapsed) { 

 nextReport = elapsed + 10.0f; 

 printf("\n\tSummary at: "); 

 fflush(stdout); 

 system("date"); // Printing a date 

 fflush(stdout); 

 printf("\n"); 

 for (size_t i = 0; i < clientErrors.size(); ++i) { 

 if (clientErrors.at(i)) 

 clientFaulty.at(i) = true; 

 clientErrors.at(i) = 0; 

 } 

 } 

 }

Which happens after 10%, 20%, ..., 90% of progress. However, if an error happens after 90% of progress - like that happened in our case - then the flag is not set and the final result will be wrong.

Trying to run gpu-burn without the nvidia drivers loaded doesn't immediately exit with a non-zero code, and console is flooded with "terminate called after..." error messages

$ ./gpu_burn 30
Burning for 30 seconds.
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >terminate called after throwing an instance of
'terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator

'
terminate called after throwing an instance of 'terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator
'
terminate called after throwing an instance of 'terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator
'
std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator
std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
'
std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
terminate called after throwing an instance of 'terminate called after throwing an instance of 'terminate called after throwing an instance of 'std::__cxx11::basic_string<ch
ar, std::char_traits, std::allocator >'
std::__cxx11::basic_string<char, std::char_traits, std::allocator >' terminate called after throwing an instance of 'terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator
'

I eventually had to CTRL-C out of this. It's on Ubuntu 22.04 with the latest gpu_burn source and cuda toolkit installed. I'm doing some bug testing of a wrapper I am using, when I hit this.

No Cuda Devices Found

Hello,

I recently build gpu-burn to test a new GPU node on our cluster. It runs and throws the following error:

./gpu_burn 
Run length not specified in the command line.  Burning for 10 secs
GPU 0: TITAN RTX (UUID: GPU-5ca89547-0a71-fc6d-f749-1651ee078536)
GPU 1: TITAN RTX (UUID: GPU-fd78f358-3e8b-e2bb-abca-a5c099d7e983)
GPU 2: TITAN RTX (UUID: GPU-e617a03a-7fe6-9418-5a58-3c91ee2f1f1d)
GPU 3: TITAN RTX (UUID: GPU-e3abf33b-367a-9704-26cc-4d97b1502511)
GPU 4: TITAN RTX (UUID: GPU-93100465-8c44-b8a0-869c-4a4fc7d50c57)
GPU 5: TITAN RTX (UUID: GPU-d9d50c3c-8fc3-f2f4-2dc8-e609b7e3075a)
GPU 6: TITAN RTX (UUID: GPU-fc8905f2-1b99-30c8-ea2c-f1b8848a3c8e)
GPU 7: TITAN RTX (UUID: GPU-5a739730-efc4-3460-ecf2-d2206dca0348)
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >'
No CUDA devices

These node have gcc 8.3 and cuda 11.0

Thanks!

It doesn't "burn" the GPUs enough

I have been recently facing some power issues in some machines with 8 Nvidia RTX 3090.

While performing the Tensorflow unit tests (when compiling tensorflow by hand) leads me consistently to kernel panics on this machine (because the power consumption is so huge, one of the gpu dies and the kernel doesn't know what to do).

I just ran gpu-burn test for a couple hours consistently, with and without doubles, with and without tensor cores. The machine runs just fine. I run tensorflow's tests again, the machine has a higher power consumption and goes dead again.

Other than that, great test for the throughput and temperature. Thanks!

How to Specify GPU?

Hello!

How can I specify one of the GPUs to run gpu-burn on it, instead of run on all the GPUs in our cluster by using "./gpu_burn"?
Could you please consider adding an option for this?

Thanks!

./gpu_burn: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by ./gpu_burn)

Hi! Thanks for the great tool
I have been trying to run it with nvidia-docker2 and that's what I get:

sudo docker run --rm --gpus all gpu_burn
./gpu_burn: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by ./gpu_burn)
./gpu_burn: /usr/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./gpu_burn)

However:

strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBC
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_3.4.20
GLIBCXX_3.4.21
GLIBCXX_3.4.22
GLIBCXX_3.4.23
GLIBCXX_3.4.24
GLIBCXX_3.4.25
GLIBCXX_3.4.26
GLIBCXX_3.4.27
GLIBCXX_3.4.28
GLIBCXX_3.4.29
GLIBCXX_3.4.30
GLIBC_2.2.5
GLIBC_2.3
GLIBC_2.14
GLIBC_2.6
GLIBC_2.33
GLIBC_2.25
GLIBC_2.18
GLIBC_2.16
GLIBC_2.32
GLIBC_2.7
GLIBC_2.3.4
GLIBC_2.17
GLIBC_2.3.2
GLIBC_2.34
GLIBC_2.4
GLIBCXX_DEBUG_MESSAGE_LENGTH

When I did try it without docker:

git clone https://github.com/wilicc/gpu-burn
cd gpu-burn
make
./gpu_burn 60

The make produced warnings:

and then the ./gpu_burn 60 shut down the system after initing the GPUs ;(

Here is an output of nvidia-smi:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A6000    Off  | 00000000:31:00.0 Off |                  Off |
| 30%   32C    P8    14W / 300W |      0MiB / 49140MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A6000    Off  | 00000000:4B:00.0 Off |                  Off |
| 30%   31C    P8     8W / 300W |      0MiB / 49140MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA RTX A6000    Off  | 00000000:B1:00.0 Off |                  Off |
| 30%   30C    P8    11W / 300W |      0MiB / 49140MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA RTX A6000    Off  | 00000000:CA:00.0 Off |                  Off |
| 30%   31C    P8     8W / 300W |      0MiB / 49140MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

And

nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

I'm on the Ubuntu 22.04.1 LTS

Any idea on what is going on here?

Problem burn Quadro RTX4000

Hello,

I have a problem when I do the burn test on my Quadro RTX4000 GPU

I feel like it doesn't really "burn" it since during the burn I can see with nvidia-smi that it only uses 45W/125W which is only one third

When I do it on my K80 GPU it uses 142/149W and 143/149W which is almost the maximum

Do I have to install some other tools to be able to have a complete burn on a RTX4000 GPU ?

Thank you

Segmentation fault on windows subsytem for linux

'''
$ ./gpu_burn
Run length not specified in the command line. Burning for 10 secs
GPU 0: NVIDIA GeForce GTX 1060 6GB (UUID: GPU-210cc47f-f02f-c51e-8d2c-4bd28141e785)
Segmentation fault

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
'''

Coudn't init a GPU test: Error in "load module"

I have compiled the tool and have ran it on a multi-gpu system. I get the following...

GPU 0: TITAN RTX (UUID: GPU-f8cb36a5-7a80-8667-e9c1-66c4acddaeaf)
GPU 1: GeForce RTX 2080 Ti (UUID: GPU-54bf72e7-8427-4d2e-da44-fafcafa8d97e)
GPU 2: GeForce RTX 2080 Ti (UUID: GPU-e2189a61-7a4e-1235-7f17-45c86780f7c8)
GPU 3: GeForce RTX 2080 Ti (UUID: GPU-d0ad5f09-7092-a5fd-10e4-a17238dbe701)
GPU 4: GeForce RTX 2080 Ti (UUID: GPU-13561097-4310-6a40-85e5-1688438e932c)
GPU 5: TITAN RTX (UUID: GPU-95425571-c209-11ed-b978-067923fd639c)
Initialized device 0 with 24190 MB of memory (23919 MB available, using 21527 MB of it), using FLOATS
Couldn't init a GPU test: Error in "load module":
100.0% proc'd: 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) errors: 0 - 0 - 0 - 0 - 0 - 0 temps: 30 C - 28 C - 26 C - 29 C - 28 C - 29 C
Summary at: Wed Jul 31 17:00:39 PDT 2019

Killing processes.. done

Tested 6 GPUs:
GPU 0: OK
GPU 1: OK
GPU 2: OK
GPU 3: OK
GPU 4: OK
GPU 5: OK

I am not sure what to do here. It says device 0 initialized but then says it failed to init the test...
The above was running the tool for 1 second but the same results for longer periods of time.

One thing I notice is that when it starts up I get a spike in GPU usage: Volatile GPU-Util for some of the GPUs hit 24% but then fall. Not sure if this can help you in any way.

Any help would be appreciated.

Support for RTX 30 series - Ampere arch

Hi,

I Downloaded the latest packages from the website and try to compile the code and give me this error

PATH=/home/bizon/.local/bin:/home/bizon/anaconda3/bin:/home/bizon/anaconda3/condabin:/usr/local/cuda-11.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:.:/usr/bin:/home/bizon/.local/bin:/home/bizon/anaconda3/bin:/home/bizon/anaconda3/condabin:/usr/local/cuda-11.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin /usr/local/cuda/bin/nvcc -I/usr/local/cuda/include -arch=compute_30 -ptx compare.cu -o compare.ptx nvcc fatal : Value 'compute_30' is not defined for option 'gpu-architecture' make: *** [Makefile:10: drv] Error 1

I tried with different options and I am using RTX 3080

make COMPUTE=8.6

Same problem, please help, thanks

support for gcc > 8

$ make -j48
...
In file included from /usr/lib/cuda/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/lib/cuda/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
make: *** [Makefile:10: drv] Error 1
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-17ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)

Can this test jetson?

How to test jetson with this

`Syntax Error near -i` when trying to execute only one GPU

I wanted to run gpu_burn on a GPU at a time.
The option -i N to execute only on GPU N, does not work.
I got a Syntax error near -i.

gpu-burn should exit early on init errors

If there is an init error on the GPU, gpu-burn does not exist but instead continues and prints "OK" at the end.

$ ./gpu_burn 3
GPU 0: GeForce GTX 1080 Ti (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
GPU 1: GeForce GTX 1080 Ti (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
GPU 2: GeForce GTX 1080 Ti (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
GPU 3: GeForce GTX 1080 Ti (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
GPU 4: GeForce GTX 1080 Ti (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
GPU 5: GeForce GTX 1080 Ti (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
GPU 6: GeForce GTX 1080 Ti (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
GPU 7: GeForce GTX 1080 Ti (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
Couldn't init a GPU test: Error in "init": CUBLAS_STATUS_NOT_INITIALIZED
100.0%  proc'd: 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s)   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 30 C - 31 C - 32 C - 29 C - 34 C - 31 C - 32 C - 30 C
        Summary at:   Wed Dec  4 18:44:29 UTC 2019

100.0%  proc'd: 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s)   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 30 C - 31 C - 32 C - 29 C - 34 C - 31 C - 32 C - 30 C
Killing processes.. done

Tested 10 GPUs:
        GPU 0: OK
        GPU 1: OK
        GPU 2: OK
        GPU 3: OK
        GPU 4: OK
        GPU 5: OK
        GPU 6: OK
        GPU 7: OK

It would be better to stop at the point where the error is thrown and not continue until the timer expired.

Dead project?

Is this not being updated anymore? I do notice that some of the forks are more updated than this one, but haven't checked all forks to find which has the most changes. Would be great if even the readme was updated to say that this project is no longer updated.

Failure during compute: Error in "Read faultyelemdata": CUDA_ERROR_UNKNOWN

Hi Wili, I have used the gpu burn to do stress test on my 2 GPUs. The Tesla K40m failed and showed the following errors. After the system stops running is shows the message "Killing process" and hangs without killing the process. I waited 20 minutes but it never stops and gives me the final summary.

I am not sure what it means and why the script is crashing. I hope you can show some light on it.

I have added all the information below.

Thanks.

OS: Centos 7.4
Cuda toolkit version: 9.0
Nvidia driver: 390.30
GCC version: 4.8.5
GPUs: Tesla K40m and Quadro M4000

$ make && ./gpu_burn -d 1200

GPU 0: Tesla K40m (UUID: GPU-660d987e-5b28-3bf6-2a2d-1d5ecae7f6df)
GPU 1: Quadro M4000 (UUID: GPU-ad447a04-f6f5-3ccf-c0f7-3218b1919de2)
Initialized device 1 with 11441 MB of memory (11330 MB available, using 10197 MB of it), using DOUBLES
Initialized device 0 with 8126 MB of memory (8016 MB available, using 7214 MB of it), using DOUBLES
10.4% proc'd: 446 (74 Gflop/s) - 6636 (423 Gflop/s) errors: 0 - 0 temps: 93 C - 57 C
Summary at: Wed 11 Apr 11:27:59 BST 2018

13.1% proc'd: 669 (74 Gflop/s) - 6952 (154 Gflop/s) errors: 0 - 0 temps: 95 C - 58 C Failure during compute: Error in "Read faultyelemdata": CUDA_ERROR_UNKNOWN
20.5% proc'd: 892 (74 Gflop/s) - -1 (154 Gflop/s) errors: 0 - -1 (DIED!) temps: 95 C - 58 C
Summary at: Wed 11 Apr 11:30:00 BST 2018

30.6% proc'd: 1561 (74 Gflop/s) - -1 (154 Gflop/s) errors: 0 - 0 (DIED!) temps: 95 C - 58 C ^[[B~5B
Summary at: Wed 11 Apr 11:32:01 BST 2018

40.7% proc'd: 2007 (74 Gflop/s) - -1 (154 Gflop/s) errors: 0 - 0 (DIED!) temps: 95 C - 58 C
Summary at: Wed 11 Apr 11:34:02 BST 2018

50.8% proc'd: 2453 (73 Gflop/s) - -1 (154 Gflop/s) errors: 0 - 0 (DIED!) temps: 95 C - 58 C
Summary at: Wed 11 Apr 11:36:03 BST 2018

60.8% proc'd: 2899 (74 Gflop/s) - -1 (154 Gflop/s) errors: 0 - 0 (DIED!) temps: 95 C - 58 C
Summary at: Wed 11 Apr 11:38:04 BST 2018

70.9% proc'd: 3568 (73 Gflop/s) - -1 (154 Gflop/s) errors: 0 - 0 (DIED!) temps: 95 C - 58 C
Summary at: Wed 11 Apr 11:40:05 BST 2018

74.0% proc'd: 3791 (73 Gflop/s) - -1 (154 Gflop/s) errors: 0 - 0 (DIED!) temps: 95 C - 58 C
Summary at: Wed 11 Apr 11:41:05 BST 2018

81.0% proc'd: 4014 (74 Gflop/s) - -1 (154 Gflop/s) errors: 0 - 0 (DIED!) temps: 95 C - 58 C
Summary at: Wed 11 Apr 11:42:06 BST 2018

91.1% proc'd: 4460 (73 Gflop/s) - -1 (154 Gflop/s) errors: 0 - 0 (DIED!) temps: 95 C - 58 C
Summary at: Wed 11 Apr 11:44:07 BST 2018

100.0% proc'd: 5129 (73 Gflop/s) - -1 (154 Gflop/s) errors: 0 - 0 (DIED!) temps: 95 C - 58 C

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 21593 C ./gpu_burn 10266MiB |
| 1 21590 C ./gpu_burn 7289MiB |
+-----------------------------------------------------------------------------+

Cuda 11.3 compatibility issue: Error in "SGEMM"

There seems to be a compatibility issue between gpu-burn and the 11.3 toolkit/465 driver.
I have replicated the behavior on two different machines, one with 2x 2080 and another with 2x 3090. Reinstalling 11.2.2 with 460 driver solves the issue.
OS: CentOS 7.9.2009 (Core) with ELRepo kernel-lt 5.4.88

CUDA Toolkit 11.3.1:

$ ./gpu_burn
Run length not specified in the command line.  Burning for 10 secs
GPU 0: NVIDIA GeForce RTX 3090 (UUID: GPU-c3832820-5d0a-f9e4-b9c8-fd92111f4b31)
GPU 1: NVIDIA GeForce RTX 3090 (UUID: GPU-ed191d51-aec3-e5bb-4a90-40c629499306)
Initialized device 1 with 24268 MB of memory (23740 MB available, using 21366 MB of it), using FLOATS
Initialized device 0 with 24268 MB of memory (23740 MB available, using 21366 MB of it), using FLOATS
Failure during compute: Error in "SGEMM": 
10.0%  proc'd: 0 (0 Gflop/s) - -1 (0 Gflop/s)   errors: 0 - -1  (DIED!)  temps: 39 C - 41 C Failure during compute: Error in "SGEMM": 
10.0%  proc'd: -1 (0 Gflop/s) - -1 (0 Gflop/s)   errors: -1  (DIED!)- -1  (DIED!)  temps: 39 C - 41 C 

No clients are alive!  Aborting

$ nvidia-smi
Thu Jun 10 11:35:38 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.19.01    Driver Version: 465.19.01    CUDA Version: 11.3     |
...

CUDA Toolkit 11.2.2:

$ ./gpu_burn 
Run length not specified in the command line.  Burning for 10 secs
GPU 0: GeForce RTX 3090 (UUID: GPU-c3832820-5d0a-f9e4-b9c8-fd92111f4b31)
GPU 1: GeForce RTX 3090 (UUID: GPU-ed191d51-aec3-e5bb-4a90-40c629499306)
Initialized device 1 with 24268 MB of memory (23736 MB available, using 21362 MB of it), using FLOATS
Initialized device 0 with 24268 MB of memory (23736 MB available, using 21362 MB of it), using FLOATS
20.0%  proc'd: 0 (0 Gflop/s) - 1333 (8237 Gflop/s)   errors: 0 - 0   temps: 39 C - 41 C 
        Summary at:   do jun 10 11:46:51 CEST 2021

40.0%  proc'd: 1333 (8165 Gflop/s) - 2666 (15162 Gflop/s)   errors: 0 - 0   temps: 39 C - 41 C 
        Summary at:   do jun 10 11:46:53 CEST 2021

70.0%  proc'd: 5332 (15355 Gflop/s) - 3999 (15209 Gflop/s)   errors: 0 - 0   temps: 53 C - 52 C 
        Summary at:   do jun 10 11:46:56 CEST 2021

90.0%  proc'd: 6665 (15408 Gflop/s) - 6665 (15168 Gflop/s)   errors: 0 - 0   temps: 53 C - 52 C 
        Summary at:   do jun 10 11:46:58 CEST 2021

100.0%  proc'd: 9331 (15338 Gflop/s) - 7998 (15186 Gflop/s)   errors: 0 - 0   temps: 55 C - 55 C 
Killing processes.. Freed memory for dev 1
Uninitted cublas
Freed memory for dev 0
Uninitted cublas
done

Tested 2 GPUs:
        GPU 0: OK
        GPU 1: OK
$ nvidia-smi
Thu Jun 10 12:04:06 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
...

gpu_burn couldn't init a GPU Test

root@cuda:~/gpu-burn-master# ./gpu_burn
Run length not specified in the command line. Burning for 10 secs
GPU 0: Tesla T4 (UUID: GPU-0aaee1d0-efc5-d3e6-a13b-f69ac8f6fe36)
Couldn't init a GPU test: Error:
0.0% proc'd: -1 (0 Gflop/s) errors: 0 (DIED!) temps: 68 C

No clients are alive! Aborting

Does it really literally destroy the gpu card if runing 24hours ?

gpu_burn always exits 0

100.0% proc'd: 18000 (6311 Gflop/s) - 12600 (6440 Gflop/s) - 11700 (6490 Gflop/s) - 11700 (6457 Gflop/s) - 11700 (6451 Gflop/s) - 10800 (6454 Gflop/s) - 10800 (6446 Gflop/s) - 10800 (6461 Gflop/s) - 10800 (6480 Gflop/s) - 10800 (6411 Gflop/s) - 10800 (6660 Gflop/s) - 11700 (6532 Gflop/s) - 10800 (6523 Gflop/s) - 10800 (6490 Gflop/s) - 10800 (6432 Gflop/s) - 10800 (6439 Gflop/s) errors: 0 - -524288000 (WARNING!)- -524288000 (WARNING!)- -524288000 (WARNING!)- -524288000 (WARNING!)- 0 - 0 - 0 - 0 - -524288000 (WARNING!)- 0 - -524288000 (WARNING!)- 0 - 0 - -524288000 (WARNING!)- 0 temps: 51 C - 45 C - 48 C - 50 C - 49 C - 46 C - 46 C - 50 C - 51 C - 47 C - 46 C - 50 C - 50 C - 47 C - 47 C - 52 C
Killing processes.. done

Tested 16 GPUs:
GPU 0: FAULTY
GPU 1: FAULTY
GPU 2: FAULTY
GPU 3: FAULTY
GPU 4: FAULTY
GPU 5: FAULTY
GPU 6: FAULTY
GPU 7: FAULTY
GPU 8: FAULTY
GPU 9: FAULTY
GPU 10: FAULTY
GPU 11: FAULTY
GPU 12: FAULTY
GPU 13: FAULTY
GPU 14: FAULTY
GPU 15: FAULTY
ubuntu@SYS-9029GP-x16GPU:/opt/gpu-burn$ echo $?
0

So looking at the source code, gpu_burn always exits 0, even when a GPU reports errors in the comparison. I modified compare.cu so that there's always an error logged, and ran it for 20 seconds and found that while all the GPUs are showing as faulty, because of injected errors, the test still exits 0.

Error in "SGEMM": CUBLAS_STATUS_INTERNAL_ERROR

Hi!
I ran gpu_burn on my work station with two Quadro RTX 8000, cuda version 10.1 and driver version 430.64.
One of the GPU died showing “Failure：Error in "SGEMM": CUBLAS_STATUS_INTERNAL_ERROR ” and got faulty finally, while the other one reported ok. So it didn't seem to be a version problem.
But I still have no idea about this error. Dose it mean there is something wrong with the GPU hardware?

Any help is appreciated!

What happen with the error log "errors: 1579 (DIED!)" ?

Hi,

The system replies "errors: 1579 (DIED!)", the log is showed below and attached.

================================
1.6% proc'd: -1 (52755 Gflop/s) errors: 1579 (DIED!) temps: 29 C Initialized device 0 with 16108 MB of memory (15422 MB available, using 13880 MB of it), using FLOATS

If anyone know what happened and the direction to solve it?
Thanks.

gpu_burn_error_log_0108.txt

!!!problems compiling gpu_burn

Hi,

I'm trying to compile gpu_burn but I'm getting some errors:

g++ -Wall -O3 -Wno-unused-result -I/usr/local/cuda-11.7/include -c gpu_burn-drv.cpp
gpu_burn-drv.cpp: In member function ‘void GPU_Test<T>::initCompareKernel()’:
gpu_burn-drv.cpp:266:92: warning: ‘CUresult cuParamSetSize(CUfunction, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetSize(d_function, __alignof(T*) + __alignof(int*) + __alignof(size_t)), "set param size");
                                                                                            ^
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14991:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetSize(CUfunction hfunc, unsigned int numbytes);
                                    ^~~~~~~~~~~~~~
gpu_burn-drv.cpp:267:61: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, 0, &d_Cdata, sizeof(T*)), "set param");
                                                             ^
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:268:82: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*), &d_faultyElemData, sizeof(T*)), "set param");
                                                                                  ^
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:269:95: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
  checkError(cuParamSetv(d_function, __alignof(T*) + __alignof(int*), &d_iters, sizeof(size_t)), "set param");
                                                                                              ^

In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:271:73: warning: ‘CUresult cuFuncSetBlockShape(CUfunction, int, int, int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuFuncSetBlockShape(d_function, g_blockSize, g_blockSize, 1), "set block size");
                                                                         ^
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14925:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuFuncSetBlockShape(CUfunction hfunc, int x, int y, int z);
                                    ^~~~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp: In member function ‘void GPU_Test<T>::compare()’:
gpu_burn-drv.cpp:276:81: warning: ‘CUresult cuLaunchGridAsync(CUfunction, int, int, CUstream)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuLaunchGridAsync(d_function, SIZE/g_blockSize, SIZE/g_blockSize, 0), "Launch grid");
                                                                                 ^
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15260:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuLaunchGridAsync(CUfunction f, int grid_width, int grid_height, CUstream hStream);
                                    ^~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp: In function ‘void listenClients(std::vector<int>, std::vector<int>, int)’:
gpu_burn-drv.cpp:471:24: warning: comparison of integer expressions of different signedness: ‘int’ and ‘long unsigned int’ [-Wsign-compare]
                 if(res < sizeof(int))
                    ~~~~^~~~~~~~~~~~~
gpu_burn-drv.cpp:484:13: warning: unused variable ‘flops’ [-Wunused-variable]
      double flops = (double)processed * (double)OPS_PER_MUL;
             ^~~~~
gpu_burn-drv.cpp:562:27: warning: comparison of integer expressions of different signedness: ‘time_t’ {aka ‘long int’} and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]
   if (startTime + runTime < thisTime)
       ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
gpu_burn-drv.cpp: In instantiation of ‘void startBurn(int, int, T*, T*, bool, bool, ssize_t) [with T = double; ssize_t = long int]’:
gpu_burn-drv.cpp:611:15:   required from ‘void launch(int, bool, bool, ssize_t) [with T = double; ssize_t = long int]’
gpu_burn-drv.cpp:741:65:   required from here
gpu_burn-drv.cpp:352:4: warning: operation on ‘eventIndex’ may be undefined [-Wsequence-point]
    eventIndex = ++eventIndex % maxEvents;
    ^~~~~~~~~~
gpu_burn-drv.cpp: In instantiation of ‘void startBurn(int, int, T*, T*, bool, bool, ssize_t) [with T = float; ssize_t = long int]’:
gpu_burn-drv.cpp:611:15:   required from ‘void launch(int, bool, bool, ssize_t) [with T = float; ssize_t = long int]’
gpu_burn-drv.cpp:743:64:   required from here
gpu_burn-drv.cpp:352:4: warning: operation on ‘eventIndex’ may be undefined [-Wsequence-point]
gpu_burn-drv.cpp: In instantiation of ‘GPU_Test<T>::GPU_Test(int, bool, bool) [with T = double]’:
gpu_burn-drv.cpp:330:9:   required from ‘void startBurn(int, int, T*, T*, bool, bool, ssize_t) [with T = double; ssize_t = long int]’
gpu_burn-drv.cpp:611:15:   required from ‘void launch(int, bool, bool, ssize_t) [with T = double; ssize_t = long int]’
gpu_burn-drv.cpp:741:65:   required from here
gpu_burn-drv.cpp:288:6: warning: ‘GPU_Test<double>::d_devNumber’ will be initialized after [-Wreorder]
  int d_devNumber;
      ^~~~~~~~~~~
gpu_burn-drv.cpp:286:7: warning:   ‘bool GPU_Test<double>::d_doubles’ [-Wreorder]
  bool d_doubles;
       ^~~~~~~~~
gpu_burn-drv.cpp:130:2: warning:   when initialized here [-Wreorder]
  GPU_Test(int dev, bool doubles, bool tensors) :
  ^~~~~~~~
gpu_burn-drv.cpp: In instantiation of ‘void GPU_Test<T>::compare() [with T = double]’:
gpu_burn-drv.cpp:349:4:   required from ‘void startBurn(int, int, T*, T*, bool, bool, ssize_t) [with T = double; ssize_t = long int]’
gpu_burn-drv.cpp:611:15:   required from ‘void launch(int, bool, bool, ssize_t) [with T = double; ssize_t = long int]’
gpu_burn-drv.cpp:741:65:   required from here
gpu_burn-drv.cpp:276:31: warning: ‘CUresult cuLaunchGridAsync(CUfunction, int, int, CUstream)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuLaunchGridAsync(d_function, SIZE/g_blockSize, SIZE/g_blockSize, 0), "Launch grid");
              ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15260:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuLaunchGridAsync(CUfunction f, int grid_width, int grid_height, CUstream hStream);
                                    ^~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:276:31: warning: ‘CUresult cuLaunchGridAsync(CUfunction, int, int, CUstream)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuLaunchGridAsync(d_function, SIZE/g_blockSize, SIZE/g_blockSize, 0), "Launch grid");
              ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15260:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuLaunchGridAsync(CUfunction f, int grid_width, int grid_height, CUstream hStream);
                                    ^~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:276:31: warning: ‘CUresult cuLaunchGridAsync(CUfunction, int, int, CUstream)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuLaunchGridAsync(d_function, SIZE/g_blockSize, SIZE/g_blockSize, 0), "Launch grid");
              ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15260:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuLaunchGridAsync(CUfunction f, int grid_width, int grid_height, CUstream hStream);
                                    ^~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:276:31: warning: ‘CUresult cuLaunchGridAsync(CUfunction, int, int, CUstream)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuLaunchGridAsync(d_function, SIZE/g_blockSize, SIZE/g_blockSize, 0), "Launch grid");
              ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15260:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuLaunchGridAsync(CUfunction f, int grid_width, int grid_height, CUstream hStream);
                                    ^~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp: In instantiation of ‘GPU_Test<T>::GPU_Test(int, bool, bool) [with T = float]’:
gpu_burn-drv.cpp:330:9:   required from ‘void startBurn(int, int, T*, T*, bool, bool, ssize_t) [with T = float; ssize_t = long int]’
gpu_burn-drv.cpp:611:15:   required from ‘void launch(int, bool, bool, ssize_t) [with T = float; ssize_t = long int]’
gpu_burn-drv.cpp:743:64:   required from here
gpu_burn-drv.cpp:288:6: warning: ‘GPU_Test<float>::d_devNumber’ will be initialized after [-Wreorder]
  int d_devNumber;
      ^~~~~~~~~~~
gpu_burn-drv.cpp:286:7: warning:   ‘bool GPU_Test<float>::d_doubles’ [-Wreorder]
  bool d_doubles;
       ^~~~~~~~~
gpu_burn-drv.cpp:130:2: warning:   when initialized here [-Wreorder]
  GPU_Test(int dev, bool doubles, bool tensors) :
  ^~~~~~~~
gpu_burn-drv.cpp: In instantiation of ‘void GPU_Test<T>::compare() [with T = float]’:
gpu_burn-drv.cpp:349:4:   required from ‘void startBurn(int, int, T*, T*, bool, bool, ssize_t) [with T = float; ssize_t = long int]’
gpu_burn-drv.cpp:611:15:   required from ‘void launch(int, bool, bool, ssize_t) [with T = float; ssize_t = long int]’
gpu_burn-drv.cpp:743:64:   required from here
gpu_burn-drv.cpp:276:31: warning: ‘CUresult cuLaunchGridAsync(CUfunction, int, int, CUstream)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuLaunchGridAsync(d_function, SIZE/g_blockSize, SIZE/g_blockSize, 0), "Launch grid");
              ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15260:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuLaunchGridAsync(CUfunction f, int grid_width, int grid_height, CUstream hStream);
                                    ^~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:276:31: warning: ‘CUresult cuLaunchGridAsync(CUfunction, int, int, CUstream)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuLaunchGridAsync(d_function, SIZE/g_blockSize, SIZE/g_blockSize, 0), "Launch grid");
              ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15260:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuLaunchGridAsync(CUfunction f, int grid_width, int grid_height, CUstream hStream);
                                    ^~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:276:31: warning: ‘CUresult cuLaunchGridAsync(CUfunction, int, int, CUstream)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuLaunchGridAsync(d_function, SIZE/g_blockSize, SIZE/g_blockSize, 0), "Launch grid");
              ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15260:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuLaunchGridAsync(CUfunction f, int grid_width, int grid_height, CUstream hStream);
                                    ^~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:276:31: warning: ‘CUresult cuLaunchGridAsync(CUfunction, int, int, CUstream)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuLaunchGridAsync(d_function, SIZE/g_blockSize, SIZE/g_blockSize, 0), "Launch grid");
              ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15260:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuLaunchGridAsync(CUfunction f, int grid_width, int grid_height, CUstream hStream);
                                    ^~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp: In instantiation of ‘void GPU_Test<T>::initCompareKernel() [with T = double]’:
gpu_burn-drv.cpp:227:3:   required from ‘void GPU_Test<T>::initBuffers(T*, T*, ssize_t) [with T = double; ssize_t = long int]’
gpu_burn-drv.cpp:331:3:   required from ‘void startBurn(int, int, T*, T*, bool, bool, ssize_t) [with T = double; ssize_t = long int]’
gpu_burn-drv.cpp:611:15:   required from ‘void launch(int, bool, bool, ssize_t) [with T = double; ssize_t = long int]’
gpu_burn-drv.cpp:741:65:   required from here
gpu_burn-drv.cpp:266:28: warning: ‘CUresult cuParamSetSize(CUfunction, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetSize(d_function, __alignof(T*) + __alignof(int*) + __alignof(size_t)), "set param size");
              ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14991:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetSize(CUfunction hfunc, unsigned int numbytes);
                                    ^~~~~~~~~~~~~~
gpu_burn-drv.cpp:266:28: warning: ‘CUresult cuParamSetSize(CUfunction, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetSize(d_function, __alignof(T*) + __alignof(int*) + __alignof(size_t)), "set param size");
              ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14991:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetSize(CUfunction hfunc, unsigned int numbytes);
                                    ^~~~~~~~~~~~~~
gpu_burn-drv.cpp:266:28: warning: ‘CUresult cuParamSetSize(CUfunction, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetSize(d_function, __alignof(T*) + __alignof(int*) + __alignof(size_t)), "set param size");
              ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14991:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetSize(CUfunction hfunc, unsigned int numbytes);
                                    ^~~~~~~~~~~~~~
gpu_burn-drv.cpp:266:28: warning: ‘CUresult cuParamSetSize(CUfunction, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetSize(d_function, __alignof(T*) + __alignof(int*) + __alignof(size_t)), "set param size");
              ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14991:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetSize(CUfunction hfunc, unsigned int numbytes);
                                    ^~~~~~~~~~~~~~
gpu_burn-drv.cpp:267:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, 0, &d_Cdata, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:267:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, 0, &d_Cdata, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:267:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, 0, &d_Cdata, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:267:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, 0, &d_Cdata, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:268:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*), &d_faultyElemData, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:268:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*), &d_faultyElemData, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:268:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*), &d_faultyElemData, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:268:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*), &d_faultyElemData, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:269:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*) + __alignof(int*), &d_iters, sizeof(size_t)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:269:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*) + __alignof(int*), &d_iters, sizeof(size_t)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:269:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*) + __alignof(int*), &d_iters, sizeof(size_t)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:269:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*) + __alignof(int*), &d_iters, sizeof(size_t)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:271:33: warning: ‘CUresult cuFuncSetBlockShape(CUfunction, int, int, int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuFuncSetBlockShape(d_function, g_blockSize, g_blockSize, 1), "set block size");
              ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14925:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuFuncSetBlockShape(CUfunction hfunc, int x, int y, int z);
                                    ^~~~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:271:33: warning: ‘CUresult cuFuncSetBlockShape(CUfunction, int, int, int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuFuncSetBlockShape(d_function, g_blockSize, g_blockSize, 1), "set block size");
              ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14925:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuFuncSetBlockShape(CUfunction hfunc, int x, int y, int z);
                                    ^~~~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:271:33: warning: ‘CUresult cuFuncSetBlockShape(CUfunction, int, int, int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuFuncSetBlockShape(d_function, g_blockSize, g_blockSize, 1), "set block size");
              ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14925:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuFuncSetBlockShape(CUfunction hfunc, int x, int y, int z);
                                    ^~~~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:271:33: warning: ‘CUresult cuFuncSetBlockShape(CUfunction, int, int, int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuFuncSetBlockShape(d_function, g_blockSize, g_blockSize, 1), "set block size");
              ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14925:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuFuncSetBlockShape(CUfunction hfunc, int x, int y, int z);
                                    ^~~~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp: In instantiation of ‘void GPU_Test<T>::initCompareKernel() [with T = float]’:
gpu_burn-drv.cpp:227:3:   required from ‘void GPU_Test<T>::initBuffers(T*, T*, ssize_t) [with T = float; ssize_t = long int]’
gpu_burn-drv.cpp:331:3:   required from ‘void startBurn(int, int, T*, T*, bool, bool, ssize_t) [with T = float; ssize_t = long int]’
gpu_burn-drv.cpp:611:15:   required from ‘void launch(int, bool, bool, ssize_t) [with T = float; ssize_t = long int]’
gpu_burn-drv.cpp:743:64:   required from here
gpu_burn-drv.cpp:266:28: warning: ‘CUresult cuParamSetSize(CUfunction, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetSize(d_function, __alignof(T*) + __alignof(int*) + __alignof(size_t)), "set param size");
              ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14991:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetSize(CUfunction hfunc, unsigned int numbytes);
                                    ^~~~~~~~~~~~~~
gpu_burn-drv.cpp:266:28: warning: ‘CUresult cuParamSetSize(CUfunction, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetSize(d_function, __alignof(T*) + __alignof(int*) + __alignof(size_t)), "set param size");
              ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14991:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetSize(CUfunction hfunc, unsigned int numbytes);
                                    ^~~~~~~~~~~~~~
gpu_burn-drv.cpp:266:28: warning: ‘CUresult cuParamSetSize(CUfunction, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetSize(d_function, __alignof(T*) + __alignof(int*) + __alignof(size_t)), "set param size");
              ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14991:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetSize(CUfunction hfunc, unsigned int numbytes);
                                    ^~~~~~~~~~~~~~
gpu_burn-drv.cpp:266:28: warning: ‘CUresult cuParamSetSize(CUfunction, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetSize(d_function, __alignof(T*) + __alignof(int*) + __alignof(size_t)), "set param size");
              ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14991:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetSize(CUfunction hfunc, unsigned int numbytes);
                                    ^~~~~~~~~~~~~~
gpu_burn-drv.cpp:267:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, 0, &d_Cdata, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:267:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, 0, &d_Cdata, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:267:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, 0, &d_Cdata, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:267:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, 0, &d_Cdata, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:268:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*), &d_faultyElemData, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:268:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*), &d_faultyElemData, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:268:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*), &d_faultyElemData, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:268:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*), &d_faultyElemData, sizeof(T*)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:269:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*) + __alignof(int*), &d_iters, sizeof(size_t)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:269:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*) + __alignof(int*), &d_iters, sizeof(size_t)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:269:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*) + __alignof(int*), &d_iters, sizeof(size_t)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:269:25: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuParamSetv(d_function, __alignof(T*) + __alignof(int*), &d_iters, sizeof(size_t)), "set param");
              ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:15092:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuParamSetv(CUfunction hfunc, int offset, void *ptr, unsigned int numbytes);
                                    ^~~~~~~~~~~
gpu_burn-drv.cpp:271:33: warning: ‘CUresult cuFuncSetBlockShape(CUfunction, int, int, int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuFuncSetBlockShape(d_function, g_blockSize, g_blockSize, 1), "set block size");
              ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14925:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuFuncSetBlockShape(CUfunction hfunc, int x, int y, int z);
                                    ^~~~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:271:33: warning: ‘CUresult cuFuncSetBlockShape(CUfunction, int, int, int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuFuncSetBlockShape(d_function, g_blockSize, g_blockSize, 1), "set block size");
              ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14925:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuFuncSetBlockShape(CUfunction hfunc, int x, int y, int z);
                                    ^~~~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:271:33: warning: ‘CUresult cuFuncSetBlockShape(CUfunction, int, int, int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuFuncSetBlockShape(d_function, g_blockSize, g_blockSize, 1), "set block size");
              ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14925:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuFuncSetBlockShape(CUfunction hfunc, int x, int y, int z);
                                    ^~~~~~~~~~~~~~~~~~~
gpu_burn-drv.cpp:271:33: warning: ‘CUresult cuFuncSetBlockShape(CUfunction, int, int, int)’ is deprecated [-Wdeprecated-declarations]
   checkError(cuFuncSetBlockShape(d_function, g_blockSize, g_blockSize, 1), "set block size");
              ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from gpu_burn-drv.cpp:51:
/usr/local/cuda-11.7/include/cuda.h:14925:36: note: declared here
 __CUDA_DEPRECATED CUresult CUDAAPI cuFuncSetBlockShape(CUfunction hfunc, int x, int y, int z);
                                    ^~~~~~~~~~~~~~~~~~~
ATH=/usr/local/cuda/bin:/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/bin:. /usr/local/cuda-11.7/bin/nvcc  -I/usr/local/cuda-11.7/include -arch=compute_50 -ptx compare.cu -o compare.ptx
g++ -o gpu_burn gpu_burn-drv.o -O3  -lcuda -L/usr/local/cuda-11.7/lib64 -L/usr/local/cuda-11.7/lib -Wl,-rpath=/usr/local/cuda-11.7/lib64 -Wl,-rpath=/usr/local/cuda-11.7/lib -lcublas -lcudart```

---
OS:
NAME="Oracle Linux Server"
VERSION="8.6"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.6"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:8:6:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8"
ORACLE_BUGZILLA_PRODUCT_VERSION=8.6
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=8.6

NVIDIA -- CUDA:
NVIDIA-SMI 515.48.07    Driver Version: 515.48.07    CUDA Version: 11.7

something that I would like to fix it

Hi.

I am using your program to verify my GPUs.

There are something inconvenient things.

There are so many logs displayed during processing. Most of all are spam to user.
I am using "CUDA_VISIBLE_DEVICES" env variable for selecting specific GPU to test. And when I select specific GPU, the temperature and GPU number are wrong.

If you don't mind, Please update it :)

Thanks.

Sub-processes segfault upon termination

gpu_burn runs just fine, but segfaults appear in the kernel log (dmesg) as soon as the program terminates

No CUDA devices

Successfully built 6099990c9287
Successfully tagged gpu_burn:latest
john@john-trx40-designare:~/gpu-burn$ docker run --rm --gpus all gpu_burn
GPU 0: TITAN RTX (UUID: GPU-9f3af471-63ff-1ccd-d82f-385cd10342d8)
GPU 1: TITAN RTX (UUID: GPU-5fa45f2a-08dd-0e1e-23a6-d5117c5018d2)
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits, std::allocator >'
No CUDA devices

My setup is to run deeplearning inside docker containers. I assume your app does the same.
Your app finds my GPUs but fails to install CUDA I assume

No issue, just thanks!

This is awesome! Just wanted to leave a note of gratitude.

GPUs temperature

Hi,

running the test on a dual GPUs system (2x1080 water cooled) I got the following result:

➜  gpu-burn git:(master) docker run --rm --gpus all gpu_burn
GPU 0: NVIDIA GeForce GTX 1080 (UUID: GPU-70f352d1-aa4f-6f8a-2033-f8dff3dca024)
GPU 1: NVIDIA GeForce GTX 1080 (UUID: GPU-8f7dc020-3610-dda7-fa8e-15cdf18a45e2)
11.7%  proc'd: 2947 (7124 Gflop/s) - 2664 (6963 Gflop/s)   errors: 0 - 0   temps: 56 C - 73 C 
	Summary at:   Sat Sep 25 17:10:59 UTC 2021

23.3%  proc'd: 5473 (7020 Gflop/s) - 5772 (6944 Gflop/s)   errors: 0 - 0   temps: 57 C - 79 C 
	Summary at:   Sat Sep 25 17:11:06 UTC 2021

35.0%  proc'd: 8420 (6951 Gflop/s) - 7992 (6853 Gflop/s)   errors: 0 - 0   temps: 59 C - 79 C 
	Summary at:   Sat Sep 25 17:11:13 UTC 2021

46.7%  proc'd: 11367 (7109 Gflop/s) - 11100 (6743 Gflop/s)   errors: 0 - 0   temps: 62 C - 82 C 
	Summary at:   Sat Sep 25 17:11:20 UTC 2021

58.3%  proc'd: 13893 (7088 Gflop/s) - 13764 (6608 Gflop/s)   errors: 0 - 0   temps: 62 C - 82 C 
	Summary at:   Sat Sep 25 17:11:27 UTC 2021

68.3%  proc'd: 16840 (7100 Gflop/s) - 15984 (6599 Gflop/s)   errors: 0 - 0   temps: 62 C - 82 C 
	Summary at:   Sat Sep 25 17:11:33 UTC 2021

80.0%  proc'd: 19366 (6981 Gflop/s) - 18648 (6485 Gflop/s)   errors: 0 - 0   temps: 63 C - 83 C 
	Summary at:   Sat Sep 25 17:11:40 UTC 2021

91.7%  proc'd: 21892 (6931 Gflop/s) - 21312 (6507 Gflop/s)   errors: 0 - 0   temps: 62 C - 80 C 
	Summary at:   Sat Sep 25 17:11:47 UTC 2021

100.0%  proc'd: 24839 (7065 Gflop/s) - 23532 (6471 Gflop/s)   errors: 0 - 0   temps: 64 C - 83 C 
Killing processes.. Initialized device 1 with 8119 MB of memory (7931 MB available, using 7138 MB of it), using FLOATS
Results are 16777216 bytes each, thus performing 444 iterations
Freed memory for dev 1
Uninitted cublas
Initialized device 0 with 8116 MB of memory (7537 MB available, using 6783 MB of it), using FLOATS
Results are 16777216 bytes each, thus performing 421 iterations
Freed memory for dev 0
Uninitted cublas
done

Tested 2 GPUs:
	GPU 0: OK
	GPU 1: OK

what could be the so-different temperatures between the 2 GPUS? is it due to the software stressing in a different way the 2 GPUs or ... my GPU#2 has some sort of thermal issue?

Makefile incompatible with cuda 11

With the new CUDA 11, compute_30 is not recognised anymore, simply replacing it with compute_50 in the Makefile was enough.

make
PATH=/home/deployer/bin:/home/deployer/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:.:/usr/bin:/home/deployer/bin:/home/deployer/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games /usr/local/cuda/bin/nvcc -I/usr/local/cuda/include -arch=compute_30 -ptx compare.cu -o compare.ptx
nvcc fatal   : Value 'compute_30' is not defined for option 'gpu-architecture'
Makefile:10: recipe for target 'drv' failed
make: *** [drv] Error 1

Edit: I originally had another issue afterwards but that was on me

Can you tell me how to specify <path to host compiler>?

Hello,

make NVCCFLAGS=-ccbin

Can you tell me how to specify ?

Thank you.

wont compile with Cuda10

for multiple GPU systems, output needs help

On some of our systems, we have 8 or more GPUs. I like this tool, but in those cases, the output wraps badly and could use a bit of formatting help. Basically, because you assume terminal width >= line length and your use of \r to return to the "beginning of the line".

Before I go implementing a solution, are you even interested?
Would a [n]curses approach be appreciated or dismissed?
Other approaches?

GPU power usage not fully utilized with 200w/350w

The RTX 3090 has a maximum power usage of 350w but gpu-burn only explored around 200w in my work-station. How can we make it fully loaded 350w/350w.

read errors and "load module" error

I'm getting various errors running gpu_burn on a newly assembled system.

GPU 0: Tesla K80 (UUID: GPU-6b15ee7a-dc27-9450-1be9-43cff0faef9b)
GPU 1: Tesla K80 (UUID: GPU-0fcbc6b5-7be5-2b6f-7f19-436f8e9a760e)
GPU 2: Tesla K80 (UUID: GPU-95c56dfe-1776-7ac4-5b28-f19ff7376cae)
GPU 3: Tesla K80 (UUID: GPU-603e0fcc-9094-13c0-b775-c15f8e2f3aa9)
Couldn't init a GPU test: Error in "load module": 
Initialized device 0 with 11441 MB of memory (11332 MB available, using 10199 MB of it), using FLOATS
Results are 16777216 bytes each, thus performing 635 iterations
read[0] error 0
50.0%  proc'd: -1 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s)   errors: 1641686072  (DIED!)- 0 - 0 - 0   temps: 33 C - 29 C - 37 C - 31 C 
	Summary at:   Sat 08 Jan 2022 06:54:33 PM EST
read[0] error 0
50.0%  proc'd: -1 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s)   errors: 797249520  (DIED!)- 0 - 0 - 0   temps: 33 C - 29 C - 37 C - 31 C read[0] error 0
50.0%  proc'd: -1 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s)   errors: 1594499040  (DIED!)- 0 - 0 - 0   temps: 33 C - 29 C - 37 C - 31 C read[0] error 0
50.0%  proc'd: -1 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s) - 0 (0 Gflop/s)   errors: -1903218736  (DIED!)- 0 - 0 - 0   temps: 33 C - 29 C - 37 C - 31 C read[0] error 0
[===== SNIP 87855 similar lines =====]
100.0%  proc'd: -1 (0 Gflop/s) - 0 (0 Gflop/s) - -1 (0 Gflop/s) - -1 (0 Gflop/s)   errors: 1720138544  (DIED!)- 0 - 1720138544  (DIED!)- 266762848  (DIED!)  temps: 33 C - 29 C - 37 C - 31 C read[0] error 0read[2] error 0read[3] error 0
100.0%  proc'd: -1 (0 Gflop/s) - 0 (0 Gflop/s) - -1 (0 Gflop/s) - -1 (0 Gflop/s)   errors: -1777579232  (DIED!)- 0 - -1777579232  (DIED!)- 1064012368  (DIED!)  temps: 33 C - 29 C - 37 C - 31 C read[0] error 0read[2] error 0read[3] error 0
100.0%  proc'd: -1 (0 Gflop/s) - 0 (0 Gflop/s) - -1 (0 Gflop/s) - -1 (0 Gflop/s)   errors: -980329712  (DIED!)- 0 - -980329712  (DIED!)- 1861261888  (DIED!)  temps: 33 C - 29 C - 37 C - 31 C 
Killing processes.. done

Tested 4 GPUs:
	GPU 0: FAULTY
	GPU 1: OK
	GPU 2: FAULTY
	GPU 3: OK

Note the snipped output. All of this output happens in the space of a second or two; it does not appear that the tests succeeded, and the "FAULTY" and "OK" results do not seem valid.

This seems potentially related to #7. I tried compiling with different values of COMPUTE as suggested there, but with similar results.

The machine has dual K80s and nvidia-smi output is below—any other specs that might be relevant?

Sun Jan  9 14:48:36 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03    Driver Version: 460.91.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000000:06:00.0 Off |                    0 |
| N/A   27C    P8    27W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           On   | 00000000:07:00.0 Off |                    0 |
| N/A   23C    P8    29W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           On   | 00000000:84:00.0 Off |                    0 |
| N/A   30C    P8    26W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           On   | 00000000:85:00.0 Off |                    0 |
| N/A   24C    P8    30W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Missing <stdexcept> header with latest commits

One of the last few commits includes a change to throw a std::runtime_error but doesn't actually include <stdexcept> so make fails depending on the environment. Some of the commonly used std library headers used to include <stdexcept> so things like std::runtime_error may have been implicitly defined.

__@__:~/cave/gpu-burn$ make
g++  -O3 -Wno-unused-result -I/usr/local/cuda/include -c gpu_burn-drv.cpp
gpu_burn-drv.cpp: In function ‘int main(int, char**)’:
gpu_burn-drv.cpp:817:23: error: ‘runtime_error’ is not a member of ‘std’
                 throw std::runtime_error("No CUDA capable GPUs found.\n");
                       ^
make: *** [gpu_burn-drv.o] Error 1

If I specify 3600, does it last for 1 hour?

Hello,

If I specify 3600, does it last for 1 hour?

When I run it, it seems to turn off after a few minutes.
Please confirm.

Thank you.

	if (nextReport < elapsed) {
	nextReport = elapsed + 10.0f;
	printf("\n\tSummary at: ");
	fflush(stdout);
	system("date"); // Printing a date
	fflush(stdout);
	printf("\n");
	for (size_t i = 0; i < clientErrors.size(); ++i) {
	if (clientErrors.at(i))
	clientFaulty.at(i) = true;
	clientErrors.at(i) = 0;
	}
	}
	}

wilicc / gpu-burn Goto Github PK

gpu-burn's Introduction

gpu-burn

Easy docker build and run

Binary packages

Building

Usage

gpu-burn's People

Contributors

Stargazers

Watchers

Forkers

gpu-burn's Issues

OS: Centos 7.4 Cuda toolkit version: 9.0 Nvidia driver: 390.30 GCC version: 4.8.5 GPUs: Tesla K40m and Quadro M4000

================================ 1.6% proc'd: -1 (52755 Gflop/s) errors: 1579 (DIED!) temps: 29 C Initialized device 0 with 16108 MB of memory (15422 MB available, using 13880 MB of it), using FLOATS

Recommend Projects

Recommend Topics

Recommend Org

Jobs

OS: Centos 7.4
Cuda toolkit version: 9.0
Nvidia driver: 390.30
GCC version: 4.8.5
GPUs: Tesla K40m and Quadro M4000

================================
1.6% proc'd: -1 (52755 Gflop/s) errors: 1579 (DIED!) temps: 29 C Initialized device 0 with 16108 MB of memory (15422 MB available, using 13880 MB of it), using FLOATS