ConnectionResetError: [Errno 104] Connection reset by peer about coperception

[dekunma]

Hi @darkdevahm, does your machine have at least 2 CPU cores available when running the training script?
If the error persists, please try setting --nworker argument to 0, which turns off multi-process data loading, but could potentially solve the problem.

[darkdevahm]

Thanks for your reply. Yes I'm using 2 CPUs with 22GB memory and Tesla V100-SXM3-32GB, so on the hardware side, I think it should be enough.
I have tried today to set the --nworkers to 0 but the issue still persists. It trains for a while (till ~33% of epoch 1), and then throws the error.

Which Pytorch version did you use?

[dekunma]

Mine was 1.11.0

[darkdevahm]

I have tried with pytorch 1.11.0, and the same error still exists.

[dekunma]

That's kind of strange...
We tested our code on Ubuntu and RHEL machines, and we also trained the models using V100 before, but we haven't encountered this issue.
I think it is some problem with PyTorch itself, but I searched online and didn't find any definite explanation or solution.
I would recommend trying to test using another machine if the error persists.

[zllxot]

I also encountered the same problem. Have you solved it? @darkdevahm