Issues about multi gpu training and sync bn about deeplab-pytorch

[kazuto1011]

I found the training parameters seem like are selected for training on a single GPU with a batch-size of 5.

If ignoring the frozen batch norm, the actual batch size is 10 since the gradients of 5 samples are accumulated twice.
The final gradients are the same if 2 samples are accumulated 5 times or are distributed on multiple GPUs.

I wonder if I want to use multi GPUs, say training with 4 GPUs, and with batch-size = 20, what would you recommend changing the learning rate?

I have no idea to tune the learning rate. DeepLab v2 paper uses the same values for different batch sizes.

And, I wonder how to enable sync batch normalization between GPUs?

Please see README. https://github.com/kazuto1011/deeplab-pytorch#training-batch-normalization