Why is there no change in multiGPU training? about contiguous_pytorch_params

[PhilJd]

Hi @lzmisscc, could you clarify what you mean by that? Are you referring to the function calls required, your training loss not going down, or something else?
Thanks!

[liuzhuang1024]

I use the usage method in the readme to call the function but there is no time reduction and run my program in a four GPUs environment.

Python 3.8.10
torch 1.6.0

[PhilJd]

Could you check the GPU usage? Maybe the job is input bound which hides the speedup? Also, how big is your model? I.e., how many paramter tensors does it contain?

[liuzhuang1024]

Model Paramter:

Total params: 1,642,048
Trainable params: 1,642,048
Non-trainable params: 0

Input size (MB): 0.73
Forward/backward pass size (MB): 1430.98
Params size (MB): 6.26
Estimated Total Size (MB): 1437.98

and the GPU usage not much change before, the input size is (96, 1000)

[PhilJd]

What's the GPU usage percentage given by nvidia-smi?

[liuzhuang1024]

What's the GPU usage percentage given by nvidia-smi?

[PhilJd]

This looks like your job is input/CPU bound, so the GPU is idle 70% of the time anyway. To see a speed up with contiguous params, you'd need to get GPU utilization closer to 100% first.

[liuzhuang1024]

ok, thanks!👍🏻

[PhilJd]

I'll close this for now, feel free to re-open if this is still an issue when GPU usage is close to 100%.