Vocabulary size is much smaller than it ought to be about wordvectors

[KevinBretonnelCohen]

An additional data point regarding the small vocabulary size that's showing up on the status message: I'm trying to run train_word2vec() on a data set that's about 1/10 the size of the data set that I was using yesterday, and it's showing the same vocabulary size--see the attached screen shot...

[KevinBretonnelCohen]

Cut the data set down by another order of magnitude--same vocabulary size showing. See screen shot.

[bmschmidt]

Huh. How are you normalizing the text? I can untar a few pubmed abstracts and run cat */*.txt | perl -pe 's/[^A-Za-z \n]/ /g;' > all.txt to get something that gives 40,195 vocab in 5.8 million words.

1. What's the output of system("head -20 YOURFILENAME | cut -c 1-80")? Does it look like real text? Or another check is; what are the first twenty rownames() of the trained object model?
2. Try rerunning install_github("bmschmidt/wordVectors"); maybe the update last week fixed it.