threads dojob waiting forever about fb.resnet.torch

[colesbury]

Make it single-threaded:

• Get rid of the body of loop()
• Replace that with the actual data loading code

[ljk628]

Try setting OMP_NUM_THREADS=1? This helps me solving #17.

[colesbury]

I'm not sure I understand. Are you running the two training runs at the same time?

[ouceduxzk]

no, run one finished and then run next one, it's sequentially

[colesbury]

If you run it again, does it hang?

[ouceduxzk]

yes, so far I tracked the problem down to the #99line in dataloader.lua threads:dojob().

[ouceduxzk]

this is perculiar since I run the code multiple times before and no problem. However, recently I have some library install and delete and environment change, which might have a influence on the low level thread library ?

[colesbury]

Maybe there's an error loading the data that's getting hidden by the threads. I'd be surprised if the issue is actually in the threads library (as opposed to just being hidden by the multi-threading)

[ouceduxzk]

what's the best way to debug the dataloader ? Right now, I print everything out to see which is not working.

Thanks

[ouceduxzk]

Thanks, man, will try it laters

[ouceduxzk]

thanks guys, actually reinstall torch and cutorch fix the problem, I do not know what happened exactly, but I learned something through this problem.

Thank you

[arthitag]

I faced a similar problem. My training code was just hanging(not showing any progress on the terminal) although the GPU memory was occupied. The GPU temperature was low(indicating inactivity). It was a multi-GPU model training with multi-threaded data loading. I faced the problem even when I tried the code on a single GPU and set the number of threads to 1.
I was getting a warning from DataparallelTable.lua that it coudn't load nccl(in the multi-GPU training). I installed nccl and also updated torch,cutorch,nn and cunn. None of this however resolved the issue.
I had to ultimately follow the suggestion of @colesbury and replace the code in the dataloader.loop() with the dataloading code. This seems to have resolved the issue and I don't see the freezing thing happening any more. Thanks! I hope someone else finds this useful.