Do i need to modify the learning rate when several gpus exploited? about py-r-fcn-multigpu

[bharatsingh430]

you need to increase learning rate when you increase number of gpus

[zengarden]

thx. More concrete situation, if i use 8gpus, lr should be 8x compared with 1gpu (same iter_size) ?

[bharatsingh430]

that worked for me, but may not always be true

[zengarden]

got it. In your coco branch, it seems that lr is still set to 1e-3 for training, while the stepsize have been set to 90000. I mean the settings in models/coco/ResNet-101/rfcn_end2end/solver_ohem.prototxt.

[bharatsingh430]

I just created this repo for multi-gpu training and it was meant for 2 GPUs with 1 iter_size on PASCAL. But I suppose, step down would be too early for coco for that. Probably I did not optimize parameters for coco when I created this repo.

The soft-nms repo contains the training schedule for ms-coco which gets 35.1 mAP, where lr is set to 0.008. But again, its dataset specific and specific to 8 GPUs.

[bharatsingh430]

I'll update this repo also in a month or so, so that master has all the features.

[zengarden]

awesome soft-nms repo. R-FCN in this repo got 30.8%, while soft-nms repo got 33.9%. i see that one difference between them is test set. COCO 2014 vs 2015 minival (but i think 2015 minival is same as 2014minival). and another difference is psroipooling. soft-nms use align psroipooling(proposed in mask-rcnn). does align pspooling improve 3.1%? i would like to reproduce the results given in soft-nms.

[bharatsingh430]

It is not completely due to mask-rcnn's roi align. I implemented what I could understand from the paper and I was seeing around 1% improvement by fixing the alignment issue. I also reduced the RPN min size from 32 to 16. Training was done till 160k iterations. Probably training longer would help more. In my experience, test-dev gives 0.2% more for R-FCN, so you should get 35.3 on test-dev.

[zengarden]

thanks a lot.

[zengarden]

I will try to reproduce soft-nms experiments.

[foralliance]

@bharatsingh430
@zengarden

I also reduced the RPN min size from 32 to 16,
does this refer to the parameters __C.TRAIN.RPN_MIN_SIZE and __C.TEST.RPN_MIN_SIZE ?
It looks like it went from 16 to 8, not from 32 to 16.

Am i right?

[bharatsingh430]

yes