Repository of pre-trained Language Models.
WARNING: a Bidirectional LM model using the MultiFiT configuration is a good model to perform text classification but with only 46 millions of parameters, it is far from being a LM that can compete with GPT-2 or BERT in NLP tasks like text generation. This my next step ;-)
Note: the training times given below are the sum of fastai Databunch creation time + model training time on 10 epochs.
I trained 1 Portuguese Bidirectional Language Model with the MultiFit configuration.
MultiFiT configuration (architecture 4 QRNN with 1550 hidden parameters by layer / tokenizer SentencePiece (15 000 tokens))
- notebook lm3-portuguese.ipynb (nbviewer of the notebook): code used to train a Portuguese Bidirectional LM on a 100 millions corpus extrated from Wikipedia by using the MultiFiT configuration.
- link to download pre-trained parameters and vocabulary in models
- notebook lm3-portuguese-classifier-olist.ipynb (nbviewer of the notebook): code used to fine-tune a Portuguese Bidirectional LM and a Sentiment Classifier on "Brazilian E-Commerce Public Dataset by Olist" dataset.
- Training with 1 NVIDIA GPU v100 on GCP
| accuracy | perplexity | training time | |
|---|---|---|---|
| forward | 39.68% | 21.76 | 8h |
| backward | 43.67% | 22.16 | 8h |
I trained 3 French Bidirectional Language Models but the best is the one trained with the MultiFit configuration.
1. MultiFiT configuration (architecture 4 QRNN with 1550 hidden parameters by layer / tokenizer SentencePiece (15 000 tokens))
- notebook lm3-french.ipynb (nbviewer of the notebook): code used to train a French Bidirectional LM on a 100 millions corpus extrated from Wikipedia by using the MultiFiT configuration.
- link to download pre-trained parameters and vocabulary in models
- notebook lm3-french-classifier-amazon.ipynb (nbviewer of the notebook): code used to fine-tune a French Bidirectional LM and a Sentiment Classifier on "French Amazon Customer Reviews" dataset.
- Training with 1 NVIDIA GPU v100 on GCP
| accuracy | perplexity | training time | |
|---|---|---|---|
| forward | 43.77% | 16.09 | 8h40 |
| backward | 49.29% | 16.58 | 8h10 |
- notebook lm2-french.ipynb (nbviewer of the notebook): code used to train a French Bidirectional LM on a 100 millions corpus extrated from Wikipedia
- link to download pre-trained parameters and vocabulary in models
- notebook lm2-french-classifier-amazon.ipynb (nbviewer of the notebook): code used to fine-tune a French Bidirectional LM and a Sentiment Classifier on "French Amazon Customer Reviews" dataset.
- Training with 1 NVIDIA GPU v100 on GCP
| accuracy | perplexity | training time | |
|---|---|---|---|
| forward | 40.99% | 19.96 | 5h30 |
| backward | 47.19% | 19.47 | 5h30 |
- notebook lm-french.ipynb (nbviewer of the notebook): code used to train a French Bidirectional LM on a 100 millions corpus extrated from Wikipedia
- link to download pre-trained parameters and vocabulary in models
- notebook lm-french-classifier-amazon.ipynb (nbviewer of the notebook): code used to fine-tune a French Bidirectional LM and a Sentiment Classifier on "French Amazon Customer Reviews" dataset.
| accuracy | perplexity | training time | |
|---|---|---|---|
| forward | 36.44% | 25.62 | 11h |
| backward | 42.65% | 27.09 | 11h |
