Terminating at Epoch 9 about relationnetworks-clevr

[fabiocarrara]

Are there any error messages when the process terminates?
Could you provide the invocation of the training script you used and the last lines of the output?

[wz337]

Thanks for the quick reply. I started a screen session running the experiment and left it overnight. When I checked this morning, the screen was terminated so I couldn't see any error message. When I rerun it again, it was giving me cuda out of memory error but tops says my process is running while the GPUs are free (but couldn't allocate anything and rebooting the machine wasn't helping). Therefore, I am wondering whether you have any idea of what the issue might be.

[wz337]

Sorry. I think it's probably the machine itself because someone else is running on it as well. Thanks for your reply and your awesome repo. I am gonna run it on another machine and I will let you know if any issue rises.