Difficult to train with LargeMargin_Softmax_Loss on cifar10 about largemargin_softmax_loss

[xqpinitial]

http://blog.csdn.net/yan_joy/article/details/53608519

[qianxinchun]

I tried "clip_gradients" the solver.prototxt, but it still ended up with 87.3365.

[xqpinitial]

firstly,please change the deplay iteration from 200 to 10 to see how the loss change.
seconfly,please reduce the base_lr = 0.0001 or lr = 0.000001 to see the loss
thirdly
1、观察数据中是否有异常样本或异常label导致数据读取异常
2、调小初始化权重，以便使softmax输入的feature尽可能变小
3、降低学习率，这样就能减小权重参数的波动范围，从而减小权重变大的可能性。这条也是网上出现较多的方法。
4、如果有BN（batch normalization）层，finetune时最好不要冻结BN的参数，否则数据分布不一致时很容易使输出值变的很大

[wy1iu]

For CIFAR10, it should be easy to train. If the network diverges, consider decreasing lambda more smoothly. Or simply lower down the difficulty of the loss, i.e. setting a smaller m.

[shenmanmiao]

Same problem with @qianxinchun , the network diverges even i set lambda_min=0.5 and m=2. @wy1iu Could you please share your training log(m=4) please?

[wy1iu]

I believe you could train it using PReLU. Using ReLU may need more parameter tuning. @shenmanmiao

[shenmanmiao]

PReLU works well on Cifar10, thanks @wy1iu for your reply.

[billhyde]

Hi,thank you for your sharing. I use CASIA-WebFace and A-softmax(Sphereface paper) train the model. The model converged and the accuracy on lfw is 97.5%. It is really hard to achieve the accuracy above 99%, It is really grateful,if you can provide any suggestions. My QQ is 729512518,

[yfllllll]

@shenmanmiao have you reproduce the result on cifar10? can you share the train_val.prototxt?