We created a new Thai sentence segmentation model called TranSentCut - Transformer Based Thai Sentence Segmentation . Here you can find code for training/evaluating the model.
After cloning the repo, cd into it and create the following directories:
- checkpoints
- data
- logs
- models
- tmp
- infer_result
Go to https://huggingface.co/airesearch/wangchanberta-base-att-spm-uncased/tree/main. Download config.json
and pytorch_model.bin
and put them in models/wangchanberta-base-att-spm-uncased
. Then download tokenizer_config.json
, sentencepiece.bpe.model
and sentencepiece.bpe.vocab
and put them in models/tokenizer
. Rename tokenizer_config.json
to config.json
. The models
directory now should look like this
models/
tokenizer/
config.json <--- tokenizer_config.json renamed
sentencepiece.bpe.model
sentencepiece.bpe.vocab
wangchanberta-base-att-spm-uncased/
config.json
pytorch_model.bin
Inside the container at <container_path>
from earlier, run
python train.py --config_path=config/TranSentCutVersion2.yaml
The result will be written to tmp/experiment_results.txt
. Model will be saved to models/version2
. The training parameters in config/TranSentCutVersion2.yaml
is the best configurations we found. It should give best f1-score (macro) of 0.0.9265. Space-correct should be 0.9626.
Once the model finished training, it can be evaluated on new data using
python eval.py --model_path=models/version2 --tokenizer_path=models/tokenizer --eval_data_path=<eval_path> --context_length=256
where <eval_path>
is the path to the evaluation data (.txt).
The trained model is also available at https://drive.google.com/drive/folders/1G29LeCn4KiW5ZJZTLn-zNaNenTgIPCSu?usp=sharing if you just want to evaluate it. Replace models/version2
in the above command with the path that you saved the model. Please get the tokenizer from https://huggingface.co/airesearch/wangchanberta-base-att-spm-uncased/tree/main. Then rename the tokenizer files and setup the working directory according to the instruction in the training section.
Once the model finished training, it can be infered on new data using
python infer.py --model_path=models/version2 --tokenizer_path=models/tokenizer --infer_data_path=<data_infer_path> --context_length=256
where <data_infer_path>
is the path to the inference data (.txt). The result will be saved to infer_result/result_<data_infer_filename>.txt
(abstract only) TranSentCut โ Transformer Based Thai Sentence Segmentation https://www.researchgate.net/publication/353996818_TranSentCut_-_Transformer_Based_Thai_Sentence_Segmentation. The full paper is under review.
- Nartawat Phong-arom 6113059
- Jirawit Sopa 6113224
- Parichaya Thanawuthikrai 6113295