CUDA driver version is insufficient for CUDA runtime version about hpc-container-maker

[samcmill]

@jianxinwang did @likueimo's suggestion of the CUDA compat library work for you? Alternatively you could downgrade the base image to CUDA 10.1.

[likueimo]

HI, I am a HPCCM user, and I just try to offer some help.
But I think your question is not about HPCCM.
And your information is not clear.
I guess that you use NGC relion and get this message ERROR: CUDA driver version is insufficient for CUDA runtime version.
In most cases is your host nvidia driver too old to support cuda version inside container.
You should upgrade your nvidia driver.

[jianxinwang]

Hi likueimo,
Thanks for your help. Yes, I'm running a singularity container for relion. According to our sysadmin,
"The version of the driver installed on the compute nodes in the GPU partition is 418.87.00. That is sufficient for CUDA versions up to and including 10.1.". I built my container from nvidia/cuda:10.2-devel-ubuntu18.04. Do you have more clues on how to get this problem solved?

[likueimo]

Only nvidia driver 440.33.01 and above native support CUDA 10.2.
nvidia driver 418.87.00 NOT native support CUDA 10.2
https://gist.github.com/ax3l/9489132

But I guess that you are running relion under HPC right?
It should be Tesla V100 or Tesla P100?
You should take a look about CUDA Compatibility for Tesla.
https://docs.nvidia.com/deploy/cuda-compatibility/index.html
https://docs.nvidia.com/datacenter/tesla/index.html

You can enable CUDA 10.2 compat library inside container.
something like this.

CUDA_HOME="/usr/local/cuda"
LD_LIBRARY_PATH=$CUDA_HOME/compat:$LD_LIBRARY_PATH

You can add above to envirmonet part of singularity definition files.
Then re-build singularity container.
singularity run --nv relion.sif should be ok now.

[jianxinwang]

Hi samcmill and likeuimo: I tried CUDA 10.1 and added the CUDA_HOME and LD_LIBRARY_PATH variable to my singularity build. The problem is gone now. But I'm having a memory error now. I have a GPU node with 178G RAM, but one MPI process still get killed. Need to figure out why on the new problem. Thanks for both of you for helping me out.