nusnlp / geccl Goto Github PK

Hello.

Thank you for releasing the code of the paper.

I have a question about the variable n_list found in label_smoothed_cross_entropy.py and new_max_margin_loss.py of GEC-PD (gec-pseudo).

It seems that each elem e_i in n_list means only the first e_i of the neg targets are used for the loss calculation. But what's its purpose?

Hi : )

I'm trying to introduce CL into CGEC (Chinese GEC) task. You used post-trained model (trained on non-native learner data) and then fine-tune (trained on native leaner data) with two strategies (NLL & CL) if I remember correctly...

I did the almost same steps use NLL strategy, but the fine-tuned model got a lower score than post-trained model in test-set (their $F_{0.5}$ scores were 5 and 25 respectively)

I think the reason might be the different data distribution, and I wanna know how you can make NLL better than DI (in paper)

I hope I made myself clear.

thanks

Hello. I reran the GEC-PD experiment with the provided data and code in the repo. However, the results I got were lower then what are reported in the repo.

Results of the repo:

S0: 41.48 | 21.44 | 34.94
S1: 31.11 | 19.37 | 27.74
G0: 42.41 | 23.01 | 36.29
G1: 32.00 | 23.28 | 29.77

S avg: 36.30 | 20.40 | 31.34
G avg: 37.21 | 23.15 | 33.03

Rerun results:

S0: 38.54 | 19.10 | 31.99
S1: 30.33 | 18.09 | 26.69
G0: 42.38 | 21.19 | 35.30
G1: 32.06 | 21.50 | 29.17

S avg: 34.43 | 18.60 | 29.34
G avg: 37.22 | 21.35 | 32.24

Environment:

• OS: Ubuntu 18.04.1 64 bits
• Python version 3.7.11
• Pytorch version 1.7.1
• CUDA Version 11.2

Here are several possible reasons I guess that led to the performance gap:

1. Choice of the best model for generating predictions with the test sets and for evaluation (calculating precision / recall / $F_{0.5}$). I used the best checkpoint during training (checkpoint_best.pt generated by fairseq). In the sample code of the repo it is checkpoint3.pt but why?

2. ERRANT version. I used errant==2.3.0.

3. Random seeds. I used [10, 20, 30] and took the average.

Since the evaluation script was not released by the repo, I am not sure how the trained models in the paper were evaluated. Could you kindly provide more details, such as releasing the evaluation script?

Thank you very much.

Hello. Thanks for your work and kindly releasing the code. Can you also provide the fine-tuning script of NLL except CL- and CL?

作者你好，请问一下该模型是否可用于中文语法纠错