Training accuracy suddenly approaches zero about alphanet

[dilinwang820]

Hi @liwei9719,

Thanks for your interest in this repo. Does this happen all the time? Would you be able to load the previous checkpoint and resume the training with expected performance? From my experiences, the training is largely stable if the gradients are clipped properly (e.g., set clip_grad_val to 1.0)

Regarding the negative training loss, I must apologize for the confusion here. The reason is that we don't use the actual alpha-divergence for logging. Instead we first compute the gradient we need, and then constructing a surrogate loss to produce this gradient. The surrogate loss might be negative. Thanks.

[liwei109]

I only trained once because of the huge cost.I have set clip_grad_val to 1.0. Does this parameter setting cause the above phenomenon?

[dilinwang820]

Maybe try to resume from a previously saved checkpoint with a different random seed?

[jun-fang]

Hi @liwei9719 @dilinwang820,

I met the same problem at around 60 epoch, and I tried multiple times to resume from a saved checkpoint with another random seed but it did not solve the issue.

@liwei9719 did you find a solution for this?

@dilinwang820 do you have an idea why this happens and what could be a good way to avoid this phenomenon?

Thanks!

[dilinwang820]

@jun-fang to my experience, the training is always stable with the default settings; maybe try to warm up a little bit with less regularization and data augmentation, and then resume with default settings?

[pprp]

I met the same problem..