Training TIme about clean-pvnet

[pengsida]

I found that dilated convolution is very slow in pytorch 1.1 with cuda 9.0.
You could try pytorch 0.4.1, which I used to train the network.

[pengsida]

gpus: https://github.com/zju3dv/clean-pvnet/blob/master/configs/linemod.yaml#L7
time: 12h33m

[kevin860213]

Thanks, did you try to train on single gpu?

[pengsida]

No, you could try it and watch the output curve.

[mentian]

gpus: https://github.com/zju3dv/clean-pvnet/blob/master/configs/linemod.yaml#L7
time: 12h33m

Hi, Sida,
I trained with your original code on "cat" model, except that I use two gpus (Titan X and 1080Ti). It takes about 50 mins for one epoch (about 20000 images, batch size=16). Would you have any suggestion on this? I assume it only takes around less than 4 mins for one epoch (240 epochs takes 12h33m) during your training.

[pengsida]

What is your pytorch version?

[mentian]

Pytorch 1.1.0 with CUDA 9.0, as indicated in your README. As far as I know, all the differences from your setting are the system version (Ubuntu 18.04), and I use 2 GPUS (one Titan X and one 1080 Ti), batch size is reduced to 16 accordingly. Then it takes around 50 mins for one epoch, which is far from twice of your time. Since I exactly follow your installation guide, except the differences I mentioned, I would really appreciate if you could suggest what might be the reason for this.

[mentian]

Thanks a lot, I would try that and let you know if I can reproduce your training time.

[mentian]

With pytorch 0.4.1, one epoch takes about 10 mins (2 GPUs). Thanks for you help.