Unstable performance about blitznet

[dvornikita]

The performance may be unstable indeed. This is a common problem of object detectors that have to solve a different problem in training and test time. NMS is not robust to small changes in boxes scores either.
Why do you mean by «step=4000 »?

[tjulyz]

@dvornikita Thanks for your reply! There is a typo for <step=4000>. It should be <step=40000>. Is it reasonable (~1%) for object detection performance using yours code? I have also observed the performance variations for semantic segmentation using BlitzNet. Is it related to 'shuffle_batch'?

[dvornikita]

--step is a parameter used in testing (how often you evaluate checkpoints). See config.py for details.
What you meant is probably --lr_decay (see code snippets on how to train the model). The learning rate is decreased twice during the training. For details check the original paper.

[tjulyz]

Thanks for your reply! I set the configurations for voc2012 as (lr_decay 25000 35000) for training as that in your paper. Have you evaluated all/some checkpoints (e.g. --step=2) when you reported your performance in your paper? If so, is the result calculated by average or max?

[dvornikita]

I see what you mean now.
We simply took the last checkpoint for each model, usually, its performance was the best or among the best (in ~0.3% range). If you train the model several times and none of the results if close enough to the reported ones it may indicate a problem, especially if it happens consistently in other settings as well. Otherwise, it's possible to observe performance fluctuating around the reported values.

[tjulyz]

Thanks a lot! I am trying to find the training problem.
Have you used warmup_lr in your training? I find the warmup_lr in the config is set as 1e-5, but the warmup_step is set to 0 as default.
If you use warmup_lr, how many steps have you set to train the network?
Thanks a lot!

[dvornikita]

@tjulyz warmup was meant to help with some other experiments. In the end, we didn’t use it in the experiments reported (that’s why it’s set to 0)
What do you mean by “training problem”?

[tjulyz]

Thanks for your kind reply!
Because I haven't reproduced the performance, I am trying to find out if there are some problems when I training the network.
I will close this issues soon. If I find the answer, I will discuss it again.
Thanks a lot!