Sentiment classification for Vietnamese text using PhoBert
This project shows how to finetune the recently released PhoBERT for sentiment classification using AIViVN's comments dataset.
The model scored 0.90849 on the public leaderboard, (winner's solution score is 0.90087):
Here we created a custom classification head on top of the BERT backbone. We concatenated the last 4 hidden representations of the [CLS]
token, which is actually <s>
in this case, and fed it to a simple MLP.
Download the competition data from https://www.aivivn.com/contests/6 . Move the *.crash
files to the ./raw
folder.
To convert the files to .csv
format, run:
$python crash2csv.py
This will create two files train.csv
and test.csv
in your ./data
folder.
Install the python bindings:
$pip3 install vncorenlp
Clone the VNCoreNLP repo: https://github.com/vncorenlp/VnCoreNLP
Follow the instructions in the original repo:
PhoBERT-base:
$wget https://public.vinai.io/PhoBERT_base_transformers.tar.gz
$tar -xzvf PhoBERT_base_transformers.tar.gz
PhoBERT-large:
$wget https://public.vinai.io/PhoBERT_large_transformers.tar.gz
$tar -xzvf PhoBERT_large_transformers.tar.gz
To perform training on a single fold, run the following command:
python train.py --fold <fold-id> \
--train_path ./data/train.csv \
--dict_path /<path-to-phobert>/dict.txt \
--config_path /<path-to-phobert>/config.json \
--bpe_codes /<path-to-phobert>/bpe.codes \
--pretrained_path /<path-to-phobert>/model.bin \
--ckpt_path ./models
--rdrsegmenter_path /<absolute-path-to>/VnCoreNLP-1.1.1.jar
Note that the rdrsegmenter_path
must be an absolute path. To fully reproduce the results, repeat for fold-id
0 to 4.
To generate the submission file, run the following command, we assume that there are 5 checkpoint named model_0.bin
to model_4.bin
in the models
folder.
python infer.py --test_path ./data/test.csv \
--dict_path /<path-to-phobert>/dict.txt \
--config_path /<path-to-phobert>/config.json \
--bpe_codes /<path-to-phobert>/bpe.codes \
--pretrained_path /<path-to-phobert>/model.bin \
--ckpt_path ./models
--rdrsegmenter_path /<absolute-path-to>/VnCoreNLP-1.1.1.jar
This will generate the submission.csv file in the current folder.