BERT pretraining sample with HuggingFace transformers.
Python 3.9+ is required.
cd /somewhere/you/like
git clone [email protected]:elm200/bert_pretrainer.git
git lfs pull
poetry install
poetry run python pretrain.py
Please read pretrain.py for details. You will be able to understand it because it is not so complicated.