prekshanema25 / answerability-metric Goto Github PK

After installation, I do pytest, but I failed to pass it.

Hi,

Thank you for sharing the code! I was trying to compute Q-BLEU1 for the task of question generation on Squad data. Could you kindly show me what is the exact command I can use to compute this score?

Below is the cmd I used which produced very low score. Also I do not understand which score is the one I am looking for. Is the Mean Answerability Score Across Questions the Q-BLEU1 score in my case?

python answerability_score.py --data_type squad --ref_file ref.txt --hyp_file pred.txt --ner_weight 0.6 --qt_weight 0.2 --re_weight 0.1 --delta 0.7 --ngram_metric Bleu_1

The output is as below:

[INFO] 2019-11-08 12:01:29,997 - answerability_score.py::compute_answerability_scores
New Score: 0.052
NER Score: 0.000
RE Score: 0.059
SW Score 0.098
QT Score: 0.000
[INFO] 2019-11-08 12:01:30,002 - answerability_score.py::compute_answerability_scores
Mean Answerability Score Across Questions: 0.235
N-gram Score: 0.378

Besides, I also tried evaluating the exactly same two files where each file only contains one line. Here is the output. Is this as expected?

python answerability_score.py --data_type squad --ref_file examples/gold.txt --hyp_file examples/pred.txt --ner_weight 0.6 --qt_weight 0.2 --re_weight 0.1 --delta 0.7 --ngram_metric Bleu_1
[INFO] 2019-11-08 13:57:00,788 - answerability_score.py::compute_answerability_scores
New Score: 0.510
NER Score: 0.000
RE Score: 0.000
SW Score 1.000
QT Score: 1.000
[INFO] 2019-11-08 13:57:00,791 - answerability_score.py::compute_answerability_scores
Mean Answerability Score Across Questions: 0.510
N-gram Score: 1.000

Many thanks!

I am confused about the example in the readme, is that command calculate just answeribility score which is mentioned in the original paper? How can I directly ge Q-BLEU1~4?