Loss is 'Nan' about anchorloss

[slryou41]

Thanks for your interest! How many classes are you using? It looks like the loss is not decreasing even when you are using warmup, which is trained with the standard cross entropy. Were you able to train with cross entropy?

[murdockhou]

Hi, 3 classes in dataset. I can trained well with cross entropy. When I use cross entropy, the first several epochs maybe also learning nothing but with training process going, the network will learned something finally. So, it's worked with cross entropy but not worked with anchor loss by using same hyper parameters.

[slryou41]

This loss picks the prediction value from the target label and modulates the loss values for the other classes, so it might fluctuate when trained with 3 classes at the beginning. I still don't understand why it goes to NaN... Do you mind adding (1-pt).clamp(min=1e-10) in line 70 to make sure the loss never sees log(0)?

[murdockhou]

No useful. I'm so sad. I trained model with crossentropy or bce all okay, only change the loss to AnchorLoss, after several epochs, got nan.......

[slryou41]

I guess the only reason why it goes to NaN is from log(0). Did you apply softmax or sigmoid before passing the output value to the loss function? This loss function assumes that the output is not normalized.

[murdockhou]

@slryou41 No, only used original output from model, not normalized. The weired is the corss entropy or bce loss is okay but anchor loss not. So I don't think that there exists error about model's output.

[slryou41]

As a sanity check, can you set gamma = 0 and see if it goes to NaN?

[murdockhou]

Useless. Meet NaN same. I gave up this try...

[jongli747]

@slryou41 Hi. thanks for sharing your awesome work. I am trying to incorporate this loss in my project. I am facing the same "nan" problem after few epoch. I have followed everything you told earlier in this issue. But nothing works for me. so I am working on binary classification and output prediction is observed from the max operation. Please tell me what to do to get the work done.