Question about calculating MAdds of dynamic network in the paper about ds-net

[changlin31]

Thanks for the valuable question.
The gate is actually very sensitive to hyperparameters. To avoid troublesome tuning of the loss balancing factors and other hyperparameters, we use different routing space for different network, e.g. we only use the slimmest few sub-networks to form the DS-Net-S.

This can be find in Appendix.A of the paper

In practice, We first test the MAdds for each sub-networks, then manually choose those sub-networks with MAdds around the target MAdds as the dynamic routing space.

[changlin31]

Yes, the MAdds are averaged over the validation set, e.g.:

DS-Net/dyn_slim/apis/train_slim_gate.py

Line 287 in 2ae9455

flops_m.update(reduced_flops.item(), input.size(0))

[sseung0703]

Thank you for your rapid answer. 😊
I think it is hard to know how much MAdds is required before the validation, but you achieved similar MAdds to comparative algorithms.
Is there any rule to achieve specific MAdds?