Test the L-Softmax loss on new dataset about largemargin_softmax_loss

[wy1iu]

What is your final loss value for m=4? Without knowing your settings, my best guess is that the m=4 loss have not been properly optimized. If your lambda is close to 0 (when the optimization is over) and the m=4 loss is reasonably low, it is highly unlikely the performance will go down.

[dldl3D]

My settings is as follows:
################## train ##################
layer {
name: "ip2"
type: "LargeMarginInnerProduct"
bottom: "bn_ip"
bottom: "label"
top: "ip2"
top: "lambda"
param {
name: "ip2"
lr_mult: 1
}
largemargin_inner_product_param {
num_output: 4
#type: SINGLE
#base: 0
type: QUADRUPLE
base: 900
gamma: 0.0000055
power: 40
iteration: 0
lambda_min: 0
weight_filler {
type: "msra"
}
}
include {
phase: TRAIN
}
}
################# test ##################
layer {
name: "ip2"
type: "LargeMarginInnerProduct"
bottom: "bn_ip"
bottom: "label"
top: "ip2"
top: "lambda"
param {
name: "ip2"
lr_mult: 0
}
largemargin_inner_product_param {
num_output: 4
type:SINGLE
base: 0
gamma: 0.0000055
power: 40
iteration: 0
lambda_min: 0
weight_filler {
type: "msra"
}
}
include {
phase: TEST
}
}
layer {
name: "accuracy"
type: "Accuracy"
bottom: "ip2"
bottom: "label"
top: "accuracy"
include {
phase: TEST
}
}
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "ip2"
bottom: "label"
top: "loss"
}
The lambda is 0.9726 when the optimization.

[wy1iu]

It seems your dataset is quite easy, since you can achieve quite good results with m=1. The final lambda is actually too high for such easy dataset. You should consider increase the parameter power or change the gamma a little bit, or even increase the iteration number in order to make lambda decrease more rapidly. I think the final lambda should be nearly 0, say below 10^-5, to achieve the best results.

[dldl3D]

When the lambda decrease rapidly, the network diverges. How can I fix it?

[wy1iu]

You should change gamma to a a smaller value, or set a small lambda_min.

[wy1iu]

BTW, I just fix a bug about lambda_min. You should use the new version