Mismatch of performance between this repository and the paper about lm-lstm-crf

[LiyuanLucasLiu]

Hi, thanks for asking. Due to a bug found in the old implementation, the results reported in the paper can not be directly compared with other papers' result (but the comparison among methods 12/13/14/15 is still valid, since they are evaluated with the same script). The comparable results are updated in the readme file, which is 91.24+/-0.12.

[jind11]

So you mean the reported scores in the paper are indeed not valid caused by a bug in the code?

[LiyuanLucasLiu]

The reported performance of the POS-tagging is valid, but the reported F1 scores are not precise (pls refer to the reported number in the ReadMe file as more accurate results).

[LiyuanLucasLiu]

I will modify the ReadMe file and highlight this information.

[jind11]

I see, could you kindly explain the bug causing this issue? I thought using the official scorer to compute the F1 score for NER should be 100% safe, isn't it? I want to make sure that my computed score using the official scorer is accurate. Many thanks!

[LiyuanLucasLiu]

Yes, the official scorer is accurate.

As to the bug, we re-implemented that script with python for better efficiency (the decode speed of the best model in our recent work is 156K words/sec, https://github.com/LiyuanLucasLiu/LD-Net). However, an elif is accidentally modified as if during the implementation refactoring.