jeffthompson / word2vecandtsne

Scripts demo-ing how to train a Word2Vec model and reduce its vector space

WORD2VEC AND TSNE

Word2Vec is cool. So is tsne. But trying to figure out how to train a model and reduce the vector space can feel really, really complicated. I definitely found that to be the case, so here are some Python scripts meant to document how to start using these great tools, and to make it easier for you to use Word2Vec and tsne in your projects. If you make something cool with this code, please let me know!

(For a more detailed tutorial on how to use these scripts, see this blog post.)

CONTENTS

• lib/: where you need to put rasterfairy and any other modules that you can't install with pip, etc
• ModelsAndData/: where the 50kCommonWords.txt file is located, and where model files will be written to by default; also includes a sample trained on H.G. Wells' Time Machine
• SampleOutput/: a sample output from a Wikipedia dump (trained including parts-of-speech), reduced to 2D
• TagTextForTraining.py: splits a text into words, then gets their part-of-speech – useful for preserving different senses of a word
• TrainModel.py: first script to run, which trains a Word2Vec model on a text file
• TsneToGrid.py: a final optional step, uses the rasterfairy module to convert the tsne vector space into an even grid
• TwoStageReduce.py: second step, takes a Word2Vec model and reduces its vector space to N dimensions
• VisualizeVectorSpace/: a Processing sketch for visualizing your flattened vector space

REQUIRED LIBRARIES

To use this code, you'll need to install some pretty hefty libraries. Luckily, they all install very easily.

• gensim for Word2Vec
• sklearn for its tsne implementation
• numpy for handling the lists of vectors
• Optional: rasterfairy for tsne-to-grid layout
• Optional: pattern for part-of-speech tagging
• Optional: Wikipedia Extractor to strip Wiki tags (if you're using a Wikipedia dump as your data source)

TO USE THIS CODE

Here's the basics, or see this blog post (if you make something cool with this code, please let me know!):

1. Install the required libraries listed above and download a data set to train on
2. Get an input text file to train on, such as a novel (a good place to start) or something larger like a Wikipedia data dump – note that larger files = much longer processing time
3. Run TrainModel.py on your text file, which outputs a binary .model file
4. Run TwoStageReduce.py on the .model file, which outputs several csv files containing your reduced model (an initial reduction run, the final run, and the final run with vectors normalized -1 to 1)
5. Optionally, run TsneToGrid.py to convert your csv file to an even grid

word2vecandtsne's Issues

Issues with new version of Gensim

Hello Jeff Thompson,

I'm trying to run your script in python but I'm running into a couple of issues.
When trying to run your TrainModel.py script I am getting this error:

ValueError: You must specify either total_examples or total_words, for proper alpha and progress calculations. The usual value is total_examples=model.corpus_count.

I am somewhat new to python so I'm having trouble understanding what is going wrong.

I just downloaded your files from github and followed the instructions, but it wont work with the sample
files your provided (TimeMachine.txt)

If you could offer me any help that would be great!

Thank you,
Arthur B. S. Rodrigues

(This is probably me not knowing python that well)

Update:

I am also running into a Syntax Error having to do with Tuple Parameter Unpacking, its not supported by
version 3 of python. Does this mean that you wrote these scripts on an earlier version of python?

incorrect link for more detailed blog

the blog link
"(For a more detailed tutorial on how to use these scripts, see ****this blog post.*****)" is invalid/ says not found

I suppose the correct link is
https://www.jeffreythompson.org/blog/2017/02/13/using-word2vec-and-tsne/

TwoStageReduce.py AttributeError: 'Word2Vec' object has no attribute 'vocab'

Jeff - I get error AttributeError: 'Word2Vec' object has no attribute 'vocab' in TwoStageReduce.py

Environment: Ubuntu 16.04.3 LTS python 2.7 gensim (3.2.0)
Steps to reproduce error:

1. modify TrainModel.py line 38 to: model.train(sentences, total_examples=model.corpus_count, epochs=model.iter) as noted in #3
2. python TrainModel.py
3. python TwoStageReduce.py

loading model...
- done
converting model/words to numpy array...
- loading 50000 most common words...
- creating list of words/vectors for reduction...
  - keeping only 50000 most common words
Traceback (most recent call last):
  File "TwoStageReduce.py", line 99, in <module>
    for word in model.vocab:
AttributeError: 'Word2Vec' object has no attribute 'vocab'