stanojevic / beer

Machine Translation Evaluation Metric

BEER 2.0

BEER 2.0 is a trained machine translation evaluation metric with high correlation with human judgment both on sentence and corpus level. For the papers that describe BEER in more detail look at the References section on this page.

This pages is about the newest version of BEER which is BEER 2.0 that was used on WMT16 metrics task. If you want to use old version from WMT15 metrics task which has some features not present in the new version (paraphrasing, reordering only evaluation, syntactic features...) then look at the old branch for expired BEER 1.1

Installation

BEER 2.0 includes all its dependencies in the tar.gz. Basic installation commands on a standard Unix terminal could be as follows:

wget https://raw.githubusercontent.com/stanojevic/beer/master/packaged/beer_2.0.tar.gz
tar xfvz beer_2.0.tar.gz

After these commands the directory beer_2.0 will contain all the necessary files.

Usage

In the installation directory there will be a shell script beer that should be used for evaluation. Its usage is straightforward:

$BEER_HOME/beer -s system_translations.en -r reference_translations.en

That command will print only the corpus level score. If sentence level score is needed then additional parameter --printSentScores will print them out.

If you want to tune the Moses system with BEER you need to compile Moses with adding the files from src_moses directory.

License

BEER is published under Apache License Version 2.0 license. In other words it's free as in "free speech", but also as in "free BEER".

This license is compatible with the libraries that BEER uses:

• liblinear-java
• snakeyaml
• scopt
• JUnit
• ScalaTest
• ScalaLibrary

Authors

Miloš Stanojević and Khalil Sima'an

Institute for Logic, Language and Computation

University of Amsterdam

References

If you use BEER please cite the first reference. Other references are more about things that are related to BEER but not about the current BEER. The new version of BEER is most similar to the one in the first paper but there are some small new things in BEER 2.0 that are not documented yet.

1. Miloš Stanojević and Khalil Sima’an -- Fitting Sentence Level Translation Evaluation with Many Dense Features - EMNLP 2014 bib
2. Miloš Stanojević and Khalil Sima’an -- BEER: BEtter Evaluation as Ranking - WMT 2014
3. Miloš Stanojević and Khalil Sima’an -- Evaluating Word Order Recursively over Permutation-Forests -- SSST 2014
4. Miloš Stanojević and Khalil Sima’an -- Evaluating MT systems with BEER -- PBML 2015
5. Miloš Stanojević and Khalil Sima’an -- BEER 1.1: ILLC UvA submission to metrics and tuning task -- WMT 2015
6. Miloš Stanojević and Khalil Sima’an -- Hierarchical Permutation Complexity for Word Order Evaluation -- COLING 2016
7. Miloš Stanojević and Khalil Sima’an -- Alternative Objective Functions for Training MT Evaluation Metrics -- ACL 2017