Why do I encounter a sudden MLM accuracy drop during training? about uer-py

[dr-GitHub-account]

Maybe the NSP accuracy remains high and stable because it is an easy task compared to MLM.

There is still a sudden rise of NSP loss, increasing from 0.010 (and less) to 0.017 in step 65700. MLM loss also increases remarkably, from 0.7 to 1.0. I think it is safe to say that both MLM and NSP suffer from a degradation. The only reason I come up with is that an error in a small fraction of the corpus occurs.