Script for output of world-level models needed about lm-lstm-crf

[LiyuanLucasLiu]

Hi achernodub, i think you mean word-level. I've completed a draft (seq_w.py), and pushed to a new branch seq_w. Since I am quite busy recently, I havnt tested it, and it could have some bugs. You are welcome to try it and make a pull request if you find bugs & fix them :-). Also, if you find it to be bug-free, feel free to let us know and close this issue :-)

[achernodub]

Hi @LiyuanLucasLiu !

You are doing a great product! Thank you a lot for a quick answer despite you are very busy, I very appreciate it.

1. I've tested seq_w.py on my side and it seems not to work properly. The main problem is that for the trained and stored model it produces different outputs (i.e. output txt files) for different launches. I did this:

python seq_w.py --load_arg exp/am_cwlm_lstm_crf.json --load_check_point exp/am_cwlm_lstm_crf.model --input_file exp/test_raw.txt --output_file output.txt --decode_type label

The results in output.txt are very different for different runs for the same input file! This is the biggest problem, I kindly ask you to fix it, if possible.

2. When I run: python eval_w.py --load_arg exp/am_cwlm_lstm_crf.json --load_check_point exp/am_cwlm_lstm_crf.model --gpu 0 --dev_file exp/dev.txt --test_file exp/test.txt

it shows an error:

Traceback (most recent call last):
  File "eval_w.py", line 93, in <module>
    print(jd['checkpoint'] + ' dev_f1: %.4f dev_rec: %.4f dev_pre: %.4f dev_acc: %.4f test_f1: %.4f test_rec: %.4f test_pre: %.4f test_acc: %.4f\n' % (dev_f1, dev_rec, dev_pre, dev_acc, test_f1, test_rec, test_pre, test_acc))
KeyError: 'checkpoint'

if I comment jd['checkpoint'], i.e. making the code in the end of file:

print(' dev_f1: %.4f dev_rec: %.4f dev_pre: %.4f dev_acc: %.4f test_f1: %.4f test_rec: %.4f test_pre: %.4f test_acc: %.4f\n' % (dev_f1, dev_rec, dev_pre, dev_acc, test_f1, test_rec, test_pre, test_acc))
    else:
        dev_acc = evaluator.calc_score(ner_model, dev_dataset_loader)
        test_acc = evaluator.calc_score(ner_model, test_dataset_loader)
        print(' dev_acc: %.4f test_acc: %.4f\n' % (dev_acc, test_acc))

But I get the same f1/accuracy results for different runs of eval_w.py!

a@aa:~/LM-LSTM-CRF ((HEAD detached at origin/seq_w))$ python eval_w.py --load_arg exp/am_cwlm_lstm_crf.json --load_check_point exp/am_cwlm_lstm_crf.model --gpu 0 --dev_file exp/dev.txt --test_file exp/test.txt 
 dev_f1: 0.3434 dev_rec: 0.3827 dev_pre: 0.3114 dev_acc: 0.5059 test_f1: 0.2709 test_rec: 0.2489 test_pre: 0.2971 test_acc: 0.4350

a@aa:~/LM-LSTM-CRF ((HEAD detached at origin/seq_w))$ python eval_w.py --load_arg exp/am_cwlm_lstm_crf.json --load_check_point exp/am_cwlm_lstm_crf.model --gpu 0 --dev_file exp/dev.txt --test_file exp/test.txt 
 dev_f1: 0.3434 dev_rec: 0.3827 dev_pre: 0.3114 dev_acc: 0.5059 test_f1: 0.2709 test_rec: 0.2489 test_pre: 0.2971 test_acc: 0.4350

a@aa:~/LM-LSTM-CRF ((HEAD detached at origin/seq_w))$ python eval_w.py --load_arg exp/am_cwlm_lstm_crf.json --load_check_point exp/am_cwlm_lstm_crf.model --gpu 0 --dev_file exp/dev.txt --test_file exp/test.txt 
 dev_f1: 0.3434 dev_rec: 0.3827 dev_pre: 0.3114 dev_acc: 0.5059 test_f1: 0.2709 test_rec: 0.2489 test_pre: 0.2971 test_acc: 0.4350

= the same for 3 runs that is with contradiction with (1), where outputs are different.

Please, help!

[LiyuanLucasLiu]

Thanks a lot! nice to hear you like our project!
I just checked the code, this is caused by a bug. As to predictor, I forgot to set ner_model to eval mode, which means the dropout is still applied during annotation, which causes unstable.
I added some code, which should fix this bug (but i havt tried it myself... ).
It would be really helpful if you can test it for us :-)

[achernodub]

@LiyuanLucasLiu yes, you really fixed it, I tested it, now it works properly. Thank you a lot! I recently prepared my own version of seq_w.py based on eval_w.py and it also worked good, now it became clear, why. Yes, the reason with dropout sounds funny! )) BTW, did you think about a variational dropout, where you freeze a neuron of RNN for the whole sequence (instead of choosing the random neuron for each time step)? https://arxiv.org/pdf/1707.09861.pdf reports the considerable increase of quality for seq-2-seq tasks, and it is intuitively clear, why this happens.

[LiyuanLucasLiu]

Thanks for letting me know. Yes, i've considered to use the variational dropout, or other kinds of dropout (like https://arxiv.org/abs/1609.08017), but i havnt tried them. Thanks for the comments, and i would tried these ideas when i have time. Also if you want to try these trickes by yourself, i'm willing to paticipate some collaboration :-)