Comments (6)
so now I ran process.py after and then
from gensim.models import Word2Vec w2v_model = Word2Vec.load("mat2vec/training/models/model_example")
and it seemed to work because I did not get an error message.
I used a word from the corpus I used to train on in the most_similar function and it said that word is not in the vocabulary. How is that possible? Also getting a lot of stop words as most similar to most words in my corpus.
from mat2vec.
To have the word in the vocabulary, it should appear more than 5 times in your corpus by default.
Option 1: expand your corpus - add more articles containing the desired word.
Option 2: change this parameter using the flag "--min_count". For example, if you want to build a vocabulary containing all words run the following:
python phrase2vec.py --corpus=data/your_corpus --model_name=your_model --min_count=1
from mat2vec.
Make sure that all the paths are correct and you do the following in the given order:
- Do all the steps from the "Set up" section of the README
- Go to mat2vec/mat2vec/training/data and create 'my_corpus' file containing abstracts (for the reference look at the content of 'corpus_example')
- Go to mat2vec/mat2vec/training and build the vocabulary for your corpus running
python phrase2vec.py --corpus=data/my_corpus --model_name=my_model
. This command takes existing file 'data/my_corpus' and generate new file 'models/my_model'. - Go to mat2vec, run python interactive shell and try some of the steps from the "Processing" section of the README but importing 'my_model', like:
w2v_my_model = Word2Vec.load("mat2vec/training/models/my_model")
from mat2vec.
To add to this, the processing needs to be done before creating the corpus file. The corpus file should contain pre-processed, tokenized abstracts, one per line. Tokens should be separated by spaces. As to how to test the model, it depends what you want to test. Some examples are in README, also check out some of the other issues in this repo which might have helpful tips.
from mat2vec.
Is it possible to load the "pretrained model" and then train it again on my custom corpus? I've tried using w2v_model = Word2Vec.load("mat2vec/training/models/pretrained_embeddings")
and then w2v_model.train('corpus_example', total_examples=w2v_model.corpus_count, epochs=20)
. But after the training I tried to find the vector for a word from my corpus and it said the word is not in the vocabulary. Also all the vectors stays the same as the previous model. What am I doing wrong?
from mat2vec.
from mat2vec.
Related Issues (20)
- No module named 'helpers' error when loading newly trained embeddings HOT 5
- Question about target and context words HOT 2
- Code used to obtain the training data and for abstract classification HOT 2
- Prediction of (new) thermoelectric materials HOT 13
- Question about the outputs HOT 5
- Request for a step by step document on how to run the code HOT 7
- Training my own word embeddings HOT 3
- I was able to train and use this on the COVID papers dataset HOT 3
- Drug repurposing for COVID HOT 3
- Script to fetch cleaned abstracts HOT 1
- my corpus is too big to be put in one large file HOT 1
- TypeError: __init__() got an unexpected keyword argument 'common_terms' HOT 10
- Training data for original model HOT 4
- About the final word embeddings. HOT 4
- Problems training the model HOT 2
- Model missing from the training folder HOT 1
- Do we need to use argument `include_extra_phrases`
- Some questions about the use of chemdataextractor tool
- Other applications HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mat2vec.