BUG: CUDA (cudaMemset) error about gpumd

[joabro]

cc @brucefan1983

[brucefan1983]

GPUMD does will choose to run on the availble GPU, starting from the smallest device ID. If you want to choose the GPU with an ID of GPU_ID, you can use $ export CUDA_VISIBLE_DEVICES=GPU_ID before submitting your GPUMD job. BTW, why you need to use cudaSetDevice? Are you integrating GPUMD into some other program? joabro <[email protected]> 于 2020年1月13日周一 17:00写道：

…

What happens? When running GPUMD on systems with multiple GPUs, the following error, sometime, occurs: CUDA Error: File: neighbor_ON1.cu Line: 222 Error code: 1 Error text: invalid argument What is the reason? As is described in this thread on the Nvidia forum <https://devtalk.nvidia.com/default/topic/525612/-invalid-argument-error-in-a-cudamemset-on-2-gpu-configuration-in-multithreaded-application/> the, most probable, reason is that: I was selecting the second GPU (device 1), allocate my memory (on device 1) on it. But my main loop was executed from another thread, and because I wasn't calling again cudaSetDevice, CUDA chose the default device (device 0) which obviously triggered an error trying to access an address of an array allocated on device 1 on device 0. It seems to be possible to solve this problem, although it might be slightly inconvenient: Since I have to call cudaSetDevice from my main loop, I need to make my main loop depend on CUDA even though a very small subset actually depend on it. Which is not ideal from a design point of view. What is the expected behaviour? It is possible to run GPUMD on systems with multiple GPUs without receiving any error messages. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#46?email_source=notifications&email_token=AF546OMR3GPIWEWWGXEACATQ5R6XTA5CNFSM4KGDTM42YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IFY3Y6Q>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF546ONYSG4BYYRLA33NXJLQ5R6XTANCNFSM4KGDTM4Q> .

[joabro]

I do not want to use cudaSetDevice, I just found this thread when I googled the problem on the internet. I am sorry that I did not make this clear, but the two blocks of text under "What is the reason" are quotes from that thread, so the words are not mine. I just thought that this might help you to solve the problem. Anyhow, I will try your suggestion and select the device ID, but this might be difficult since I do not know which of the two GPUs that I have been allocated. I think I will have forward this question to the support staff for the computer cluster so that they can tell me how to write the submit scripts. I guess the worst case scenario is that I would need to request both GPUs and then run the simulation on only one of them, which has the drawback that the calculations will remain longer in the queue and require double as many core hours.

________________________________ From: Zheyong Fan <[email protected]> Sent: Monday, January 13, 2020 4:45:55 PM To: brucefan1983/GPUMD <[email protected]> Cc: joabro <[email protected]>; Author <[email protected]> Subject: Re: [brucefan1983/GPUMD] BUG: CUDA error on systems with multiple GPUs (#46) GPUMD does will choose to run on the availble GPU, starting from the smallest device ID. If you want to choose the GPU with an ID of GPU_ID, you can use $ export CUDA_VISIBLE_DEVICES=GPU_ID before submitting your GPUMD job. BTW, why you need to use cudaSetDevice? Are you integrating GPUMD into some other program? joabro <[email protected]> 于 2020年1月13日周一 17:00写道：

What happens? When running GPUMD on systems with multiple GPUs, the following error, sometime, occurs: CUDA Error: File: neighbor_ON1.cu Line: 222 Error code: 1 Error text: invalid argument What is the reason? As is described in this thread on the Nvidia forum <https://devtalk.nvidia.com/default/topic/525612/-invalid-argument-error-in-a-cudamemset-on-2-gpu-configuration-in-multithreaded-application/> the, most probable, reason is that: I was selecting the second GPU (device 1), allocate my memory (on device 1) on it. But my main loop was executed from another thread, and because I wasn't calling again cudaSetDevice, CUDA chose the default device (device 0) which obviously triggered an error trying to access an address of an array allocated on device 1 on device 0. It seems to be possible to solve this problem, although it might be slightly inconvenient: Since I have to call cudaSetDevice from my main loop, I need to make my main loop depend on CUDA even though a very small subset actually depend on it. Which is not ideal from a design point of view. What is the expected behaviour? It is possible to run GPUMD on systems with multiple GPUs without receiving any error messages. ― You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#46?email_source=notifications&email_token=AF546OMR3GPIWEWWGXEACATQ5R6XTA5CNFSM4KGDTM42YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IFY3Y6Q>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF546ONYSG4BYYRLA33NXJLQ5R6XTANCNFSM4KGDTM4Q> .

― You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#46?email_source=notifications&email_token=AN6FUI35O7H7EUEBXYZAVJDQ5SEDHA5CNFSM4KGDTM42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIZGKRY#issuecomment-573728071>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AN6FUI35RFVEGJQVBNQVRLDQ5SEDHANCNFSM4KGDTM4Q>.

[brucefan1983]

You can try to run the command $ nvidia-smi to check the IDs of the GPUs and what GPUs are busy. Apart from selecting the GPU for use before running GPUMD, you can also modify the code: just adding cudaSetDevice(your_gpu_id); in the begining of the main function in main.cu. Zheyong joabro <[email protected]> 于 2020年1月13日周一 18:35写道：

…

I do not want to use cudaSetDevice, I just found this thread when I googled the problem on the internet. I am sorry that I did not make this clear, but the two blocks of text under "What is the reason" are quotes from that thread, so the words are not mine. I just thought that this might help you to solve the problem. Anyhow, I will try your suggestion and select the device ID, but this might be difficult since I do not know which of the two GPUs that I have been allocated. I think I will have forward this question to the support staff for the computer cluster so that they can tell me how to write the submit scripts. I guess the worst case scenario is that I would need to request both GPUs and then run the simulation on only one of them, which has the drawback that the calculations will remain longer in the queue and require double as many core hours. ________________________________ From: Zheyong Fan ***@***.***> Sent: Monday, January 13, 2020 4:45:55 PM To: brucefan1983/GPUMD ***@***.***> Cc: joabro ***@***.***>; Author ***@***.***> Subject: Re: [brucefan1983/GPUMD] BUG: CUDA error on systems with multiple GPUs (#46) GPUMD does will choose to run on the availble GPU, starting from the smallest device ID. If you want to choose the GPU with an ID of GPU_ID, you can use $ export CUDA_VISIBLE_DEVICES=GPU_ID before submitting your GPUMD job. BTW, why you need to use cudaSetDevice? Are you integrating GPUMD into some other program? joabro ***@***.***> 于 2020年1月13日周一 17:00写道： > What happens? > > When running GPUMD on systems with multiple GPUs, the following error, > sometime, occurs: > > CUDA Error: > File: neighbor_ON1.cu > Line: 222 > Error code: 1 > Error text: invalid argument > > What is the reason? > > As is described in this thread on the Nvidia forum > < https://devtalk.nvidia.com/default/topic/525612/-invalid-argument-error-in-a-cudamemset-on-2-gpu-configuration-in-multithreaded-application/> > the, most probable, reason is that: > > I was selecting the second GPU (device 1), allocate my memory (on device > 1) on it. But my main loop was executed from another thread, and because I > wasn't calling again cudaSetDevice, CUDA chose the default device (device > 0) which obviously triggered an error trying to access an address of an > array allocated on device 1 on device 0. > > It seems to be possible to solve this problem, although it might be > slightly inconvenient: > > Since I have to call cudaSetDevice from my main loop, I need to make my > main loop depend on CUDA even though a very small subset actually depend on > it. Which is not ideal from a design point of view. > > What is the expected behaviour? > > It is possible to run GPUMD on systems with multiple GPUs without > receiving any error messages. > > ― > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > < #46?email_source=notifications&email_token=AF546OMR3GPIWEWWGXEACATQ5R6XTA5CNFSM4KGDTM42YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IFY3Y6Q>, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AF546ONYSG4BYYRLA33NXJLQ5R6XTANCNFSM4KGDTM4Q> > . > ― You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub< #46?email_source=notifications&email_token=AN6FUI35O7H7EUEBXYZAVJDQ5SEDHA5CNFSM4KGDTM42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIZGKRY#issuecomment-573728071>, or unsubscribe< https://github.com/notifications/unsubscribe-auth/AN6FUI35RFVEGJQVBNQVRLDQ5SEDHANCNFSM4KGDTM4Q>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#46?email_source=notifications&email_token=AF546OPLW2FUPINVQDVYMODQ5SJ5NA5CNFSM4KGDTM42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIZMO6I#issuecomment-573753209>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF546OPBJLUR2LWZOR42U2LQ5SJ5NANCNFSM4KGDTM4Q> .

[joabro]

@brucefan1983 I have just found out that this problem occurs even though I am running the simulation on a node with a single GPU. I will change the title and description accordingly.

[brucefan1983]

Could you send me the input files? 2nd order is enough.

…

On Tue, Jan 14, 2020 at 4:09 AM joabro ***@***.***> wrote: @brucefan1983 <https://github.com/brucefan1983> I have just found out that this problem occurs even though I am running the simulation on a node with a single GPU. I will change the title and description accordingly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#46?email_source=notifications&email_token=AF546ONTV3E6T4JTIQ43EB3Q5TDAPA5CNFSM4KGDTM42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEI2DQSY#issuecomment-573847627>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF546OMBMFYXGAPUUQC7QRTQ5TDAPANCNFSM4KGDTM4Q> .

[joabro]

Here are the input files:
test_vel100_nvt_lan_T100_Tc200_dump1000_run100000.tar.gz
As well as the potential:
ols_original_model_1_2x2x2_orders_2-3_tol1e-08.tar.gz

[brucefan1983]

Thanks for your inputs. I have solved the problem. It is bug in the code related to the neighbor list construction/update. Some arrays only have a length of N (number of atoms), but when the cutoff distance is 0.1 Angstrom (I recommended this to you, sorry), the number of bins exceeds N, which results in some invalid memory access. For the time being, I have disabled neighbor list construction/update for the FCP potential (need -DUSE_FCP in the makefile), becasue we don't need the neighbor list at all for this potential.

For your inputs, I would like to mention that, to use NetCDF, you need to build it according to the manual:

https://gpumd.zheyongfan.org/index.php/NetCDF_setup
https://gpumd.zheyongfan.org/index.php/The_dump_position_keyword

Without the compute command, your inputs run correctly in my laptop now.

Zheyong

[brucefan1983]

Sorry, I have a typo,
Without the compute command
should be
Without the dump_position command

[joabro]

Thank for the taking care of this so quickly @brucefan1983. I have now been able to run all the calculations without any issues.

By the way, I have already installed netcdf according to the instructions that you refer to. It would, however, be great if GPUMD could have support for other, especially more recent, versions of netcdf. Is this something that you are working on?

I suppose it should be possible to close this issue, right?

[brucefan1983]

The netcdf implementation was created and maintained by Alex [email protected] He might be interested in knowing your suggestions. Zheyong joabro <[email protected]> 于 2020年1月14日周二 13:15写道：

…

Thank for the taking care of this so quickly @brucefan1983 <https://github.com/brucefan1983>. I have now been able to run all the calculations without any issues. By the way, I have already installed netcdf according to the instructions that you refer to. It would, however, be great if GPUMD could have support for other, especially more recent, versions of netcdf. Is this something that you are working on? I suppose it should be possible to close this issue, right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#46?email_source=notifications&email_token=AF546OIV5XIKLYI4APW2SS3Q5WNETA5CNFSM4KGDTM42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEI4HQSQ#issuecomment-574126154>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF546ON5AFUZ4JBYN4EXPA3Q5WNETANCNFSM4KGDTM4Q> .

[brucefan1983]

Since this bug was fixed I close this issue now.