Very large d_loss about pro_gan_pytorch

[akanimax]

@jiujing23333,

I presume you are using a multi-gpu setup here.
Yes, this is currently a know issue (bug) with the package. The MinBatchStd layer is not properly optimized to take care of the data-parallel Multi-GPU setup. There is currently no synchronization taking place for the divided minibatch to obtain the proper minibatchstd values back at GPU_1. I am working on a solution which just involves removing the last layer of the discriminator out of the dataParallel block so that everything synchronizes before the last layer's calculations are performed. But this is a temporary solution since, it makes the training way slower and consumes very little of the other parallel GPUs.

For now, you could either ignore the very high loss values produced by only the discriminator [they will go even bigger 😆], because the GAN trains fine. Or, you could run the code on only one gpu.

Please feel free to ask if you have any more questions.

Also, please feel free to suggest if you have a better solution for this problem.

Cheers 🍻!
@akanimax

[jiujing23333]

@akanimax
Very thanks, it really works. Now I have another problem, the parameter beta of EMA is set as 0 in your code, which means EMA is always closed. Why?

[jiujing23333]

@akanimax
I'm so sorry. I notice that the zero beta is only for init...