ConnectionResetError: [Errno 104] Connection reset by peer AND ValueError: Expected input batch_size (423) to match target batch_size (465). about upsnet

[YuwenXiong]

This error means you encounter training failure (usually because lr is too large), please note that upsnet_resnet50_coco.yaml is for 16 GPUs w/ Horovod, if you use that config w/o Horovod then you will only use 1 GPU and lr will be too large (however due to the warmup you may not encounter that issue at the beginning). If that is the case, please use upsnet_resnet50_coco_4gpu.yaml instead.

[Xiaolongbetter]

Thanks for your patience. The truth seems to be true. When I only use 1 GPU with upsnet_resnet50_coco.yaml, this error will appear around batch[300]. When I use 2 GPUs with upsnet_resnet50_coco_4gpu.yaml, this error will appear around batch[12000]. If I want to use two GPU training, can I guarantee the normal operation by reducing the lr in upsnet_resnet50_coco_4gpu.yaml, or I need to have at least 4 GPUs?

[YuwenXiong]

Then please double #iter and reduce lr by 2x

[Xiaolongbetter]

Thank you very much for your answer.