This repo involves the following components:
src/bert_utils.py
: utility to load pretrained BERT model and test that the output is the same as the implemented modelsrc/bert.ipynb
:- Implement BERT modules from scratch using PyTorch including attention and embedding modules
- Finetune BERT on IMDB dataset for sentiment classification task
- Train BERT on WikiText2 dataset for masked LM task