cuda error an illegal memory access was encountered about lbpm

[JamesEMcClure]

I have not tried this GPU version but I think it should work in principle -- can you share the cmake configure step here, especially the CUDA compile line. For GTX1060 I think it should look something like this

    -D CMAKE_CUDA_FLAGS="-arch sm_61"   \

Also, re you launching this with one MPI process per GPU or in some other way? I am not sure if four processes per GPU will work unless cuda MPS is running on your machine.

Sometimes I've seen funny errors if the GPU runs out of memory, so it could be worth confirming that this isn't happening here. Once this is ruled out, the next thing to do is to build with debugging symbols and launch with cuda-memcheck for a single GPU run

mpirun -np 1 cuda-memcheck $LBPM_DIR/lbpm_color_simulator input.db

This should quickly narrow down the location of the illegal memory access.

[thomaram]

The command:
mpirun -np 1 cuda-memcheck $LBPM_DIR/lbpm_color_simulator input.db

gave the following warning:
Warning: Failed to start MPI with necessary thread support, thread support will be disabled

Is this causing the issue?

[JamesEMcClure]

No, this just means that MPI_THREAD_MULTIPLE is not enabled (either because it isn't turned on or because the MPI library you are using doesn't support it). This just means the analysis threads will be serialized, and should not impact the results.

[thomaram]

Ok. Here is the rest of the pipe:

Stack Trace:
0x00000000004396ea: lbpm_color_simulator
0x000000000043a2c9: lbpm_color_simulator StackTrace::backtrace()
0x000000000043d411: lbpm_color_simulator StackTrace::getCallStack()
0x000000000042cc2d: lbpm_color_simulator Utilities::abort(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)
0x0000000000426823: lbpm_color_simulator Domain::initialize(std::shared_ptr)
0x0000000000427133: lbpm_color_simulator Domain::Domain(std::shared_ptr, ompi_communicator_t*)
0x0000000000421bcc: lbpm_color_simulator ScaLBL_ColorModel::SetDomain()
0x0000000000419c4f: lbpm_color_simulator main
0x00007fcca5873830: libc.so.6 __libc_start_main libc-start.c:325
0x000000000041aea9: lbpm_color_simulator _start

Failed insist: nprocs == nproc[0]*nproc[1]*nproc[2]
Message: Fatal error in processor count!

[thomaram]

After amending the input.db and run on only one GPU. The build ran. I therefore close this issue