About the Super Huge Training Cost about flownet2-pytorch

[PkuRainBow]

@fitsumreda

[fitsumreda]

@PkuRainBow
the default total_num_epochs is there as default.

To reproduce the paper results, you'll only need to run (600K (Short), 1.2M (Long), and 1.7M (Fine)) mini-batch iterations (not epochs).
so, with the 1.5s/min-batch that you are getting, it would take almost 10 days (1.5 * 6e5/ 3600.) on a single GPU to finish Short .

This is however much faster for FlowNet-S, it would only take 24 hours on a single GPU.

if you want to reproduce FlowNet2 results, FlowNet-C needs to be trained only once. The remaining networks (FlowNet-Fusion and FlowNet-SD) don't use the correlation layer and can be trained efficiently.

There are still rooms for further optimisation of the Correlation kernel (which is the bottleneck). We may use optimized linear algebra libraries.

[PkuRainBow]

@fitsumreda So according to your advice, 600K mini-batches means 200 epoch over flying chairs as every epoch will takes about 3000 mini-batches.

[fitsumreda]

right.

[PkuRainBow]

@fitsumreda Thanks for your advice! FlowNetS is really fast, besides, it would be great if you could share whether could your implementation reproduce the reported numbers in FlowNet-v1 of FlowNetS.

According to my current estimate, it will takes at most 4 hours to train the FlowNetS for 200 epoches.

[fitsumreda]

it gets close results on FlowNet-S, but not able to quite reproduce the paper results
since the data augmentation part is missing, the networks are most likely overfitting

[MatthewD1993]

@fitsumreda Could you share how you train FlowNet2C? I trained it for about the 10 epochs but the result looks ugly. The EPE error does not have a trend of decreasing. While FlowNet2S gives reasonable result after just 1 epoch. Thanks!