mfaruqui / eval-word-vectors

Easy to use scripts for evaluating word vectors on a variety of tasks.

eval-word-vectors

Manaal Faruqui, [email protected]

Easy-to-use scripts for evaluating word vectors on a variety of tasks. These are the scripts that run behind the online tool on http://www.wordvectors.org/. I will be adding more evaluation scripts here over the course of time.

Requirements

1. Python 2.7 (+numpy package)

Data you need

1. Word vector file
2. Any word similarity evaluation file (if you are not using the provided ones)

Each vector file should have one word vector per line as follows (space delimited):-

the -1.0 2.4 -0.3 ...

Evaluating on multiple word sim tasks

python all_wordsim.py word_vec_file word_sim_file_dir

python all_wordsim.py skip-gram-vecs.txt data/word-sim/

Evaluating on one word sim task

python wordsim.py word_vec_file word_sim_file

word_sim_file should be in the same format as files in data/word-sim/

Reference

Please make sure to cite the papers corresponding to the word similarity dataset that you are using. This list of citation can be found at http://www.wordvectors.org/.

Please cite the following paper if you use this tool:

@InProceedings{faruqui-2014:SystemDemo,
  author    = {Faruqui, Manaal  and  Dyer, Chris},
  title     = {Community Evaluation and Exchange of Word Vectors at wordvectors.org},
  booktitle = {Proceedings of ACL: System Demonstrations},
  year      = {2014},
}

eval-word-vectors's Issues

Incorrect evaluation result

I get strange results when evaluating this model:
http://vectors.nlpl.eu/repository/10/11.zip

Serial              Dataset       Num Pairs       Not found             Rho

 1         EN-MC-30.txt              30               0         -0.1146
 2     EN-MEN-TR-3k.txt            3000               5          0.2697
 3     EN-MTurk-287.txt             287              10          0.3212
 4     EN-MTurk-771.txt             771               3          0.1994
 5         EN-RG-65.txt              65               0         -0.0006
 6   EN-RW-STANFORD.txt            2034             947          0.3325
 7    EN-SIMLEX-999.txt             999               2         -0.0299
 8      EN-VERB-143.txt             144              58          0.0305
 9    EN-WS-353-ALL.txt             353               0          0.2661
10    EN-WS-353-REL.txt             252               0          0.2858
11    EN-WS-353-SIM.txt             203               0          0.2514
12        EN-YP-130.txt             130               0          0.1524

The SimLex-999 score is negative, while gensim evaluation of the same model yields a score of 0.4189 with SimLex-999. Am I doing something wrong, or is there a bug somewhere?

Missing SimVerb-3500 file

The site wordvectors.org contains an evaluation for SimVerb-3500 (Gerz et al., 2016) however the data directory in this repository does not seem to contain the data for this evaluation. Can this be added?

Open source?

Hello!

I am just wondering whether these tools either are or are intended to be open source, and if possible, you could add a license stating as much could be added to the page? As it is right now, I am not legally allowed to make changes and publish them, which is sad, as these tools are really great!

Regards,
Kjetil Bugge Kristoffersen