rillaha / baselines-emnlp2016

This repository contains baseline models, training scripts, and instructions on how to reproduce our results for our state-of-art grammar correction system from M. Junczys-Dowmunt, R. Grundkiewicz: Phrase-based Machine Translation is State-of-the-Art for Automatic Grammatical Error Correction, EMNLP 2016.

Phrase-based Machine Translation is State-of-the-Art for Automatic Grammatical Error Correction

This repository contains baseline models, training scripts, and instructions on how to reproduce our results for our state-of-art grammar correction system from M. Junczys-Dowmunt, R. Grundkiewicz: Phrase-based Machine Translation is State-of-the-Art for Automatic Grammatical Error Correction, EMNLP 2016.

Citation

@InProceedings{junczysdowmunt-grundkiewicz:2016:EMNLP2016,
  author    = {Junczys-Dowmunt, Marcin  and  Grundkiewicz, Roman},
  title     = {Phrase-based Machine Translation is State-of-the-Art for
               Automatic Grammatical Error Correction},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in
               Natural Language Processing},
  month     = {November},
  year      = {2016},
  address   = {Austin, Texas},
  publisher = {Association for Computational Linguistics},
  pages     = {1546--1556},
  url       = {https://aclweb.org/anthology/D16-1161}
}

Update

The results reported in the camera-ready version of the paper on the dataset from Bryant and Ng (2015) (Tab. 4, three last columns) are understated due to the invalid preparation of the M2 file. The correct scores are as follows:

System	Prec.	Recall	M^2
Baseline	69.22	37.00	58.95
+CCLM	76.66	36.39	62.77
Best dense	71.11	37.44	60.27
+CCLM	79.76	39.52	66.27
Best sparse	76.48	35.99	62.43
+CCLM	80.57	39.74	66.83

We would like to thank Shamil Chollampatt for reporting this issue!

Outputs

Outputs generated by our models for the CoNLL-2014 test set are available in the folder outputs. These correspond to Table 4 of our paper. See the README in that folder for more information.

Baseline models

You can download and run our baseline models (558M).

models/
├── data
│   ├── lm.cor.kenlm
│   ├── osm.kenlm
│   └── phrase-table.0-0.gz
├── moses.dense-cclm.mert.avg.ini
├── moses.dense.mert.avg.ini
├── moses.sparse-cclm.mert.avg.ini
├── moses.sparse.mert.avg.ini
└── sparse
    ├── moses.cc.sparse
    └── moses.wiki.sparse

The four configuration *.ini files corresponds to the last four systems described in Table 4.

To use the models you have to install Moses decoder (branch master) It has to be compiled with support for 9-gram kenLM language models, e.g.:

/usr/bin/bjam -j16 --max-kenlm-order=9

The language model data are available in separate packages:

• Wikipedia language model (22G)
• Common Crawl language model (26G)

The packages contain:

wikilm/
├── wiki.blm
├── wiki.classes.gz
└── wiki.wclm.kenlm
cclm/
├── cc.classes.gz
├── cc.kenlm
└── cc.wclm.kenlm

Adjust absolute paths in moses.*.ini files. You can do this by replacing /path/to/ with the path to the directory where you downloaded models and language models. Finally, run moses, e.g.:

/path/to/mosesdecoder/bin/moses -f moses.dense.mert.avg.ini < input.txt

The input file should contain one sentence per line and each sentence has to follow the Moses tokenization.

Training models

Training is described in the README in the folder train.