prashantranjan09 / wordembeddings-elmo-fasttext-word2vec

Using pre trained word embeddings (Fasttext, Word2Vec)

WordEmbeddings-ELMo, Fasttext, FastText (Gensim) and Word2Vec

This implementation gives the flexibility of choosing word embeddings on your corpus. One has the option of choosing word Embeddings from ELMo (https://arxiv.org/pdf/1802.05365.pdf) - recently introduced by Allennlp and these word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. Also fastext embeddings (https://arxiv.org/pdf/1712.09405.pdf) published in LREC from Thomas Mikolov and team is available. ELMo embeddings outperformed the Fastext, Glove and Word2Vec on an average by 2~2.5% on a simple Imdb sentiment classification task (Keras Dataset).

USAGE:

To run it on the Imdb dataset,

run: python main.py

To run it on your data: comment out line 32-40 and uncomment 41-53

FILES:

• word_embeddings.py – contains all the functions for embedding and choosing which word embedding model you want to choose.
• config.json – you can mention all your parameters here (embedding dimension, maxlen for padding, etc)
• model_params.json - you can mention all your model parameters here (epochs, batch size etc.)
• main.py – This is the main file. Just use this file to run in terminal.

You have the option of choosing the word vector model

In config.json specify “option” as 0 – Word2vec, 1 – Gensim FastText, 2- Fasttext (FAIR), 3- ELMo

The model is very generic. You can change your model as per your requirements.

Feel free to reach out in case you need any help.

Special thanks to Jacob Zweig for the write up: https://towardsdatascience.com/elmo-embeddings-in-keras-with-tensorflow-hub-7eb6f0145440. Its a good 2 min read.

wordembeddings-elmo-fasttext-word2vec's Issues

Elmo embeddings with different dataset

Hello, i tried to run your code with my dataset, i am able to word2vec/fasttext gensim with my dataset but unable to understand how u r handling word vectors for Elmo. in below screenshot.

how to change these values in below elmo function ( in my case x_text, y is a X Y form )
(train_x,train_y,test_x,test_y,max_len)

please advise how to change

Error when running code

hello , i get below mentioned error when i tried to train model with W2V, Fastttext

please advise. thnx

elmo

1

Elmo embeddings for urdu language corpus

Do we have to change anything other then dataset to train elmo embedding in urdu language corpus?is it neccessary to use GPU to train model and any preprocessing of data required before training ?

"https://tfhub.dev/google/elom/1" can not open

Allocation of 1258291200 exceeds 10% of system memory. Killed

Hi,

I get the following error when I run the code. Any workaround for this?

2019-01-25 18:56:48.821520: W tensorflow/core/framework/allocator.cc:101] Allocation of 18454937600 exceeds 10% of system memory.
2019-01-25 18:56:48.821663: W tensorflow/core/framework/allocator.cc:101] Allocation of 1258291200 exceeds 10% of system memory.
Killed
.

I have already tried decreasing batch size but didn't help.

How to handle out of vocabulary words while using pretrained embeddings?

Hi everyone,

I would like to use German align vectors to train a crosslingual classification model (https://fasttext.cc/docs/en/aligned-vectors.html). If I got it right, wiki.de.align.vec is the file I need? I tried to implement it using word2vec_format, but ran into a problem of out of vocabulary words.

I know that this problem has been solved by fasttext. It obtaines semantic similar word vectors by breaking words into ngrams and it sounds awesome. To load the fasttext model I need a .bin file, which is not provided for aligned vectors. Do you have any ideas of solving the problem?

Thank you!