gbswt5's Introduction

GBSWT5

Huggingface Transformers-compatible GBST+T5 implementation(as CharFormer(Tay et al., 2022)) for GBST-KEByT5 Model.

Supports following pretrained checkpoints:

etri-lirs/gbst-kebyt5-base-preview
etri-lirs/gbst-kebyt5-large-preview (not yet)

How To Use

Install with pip.

pip install git+https://github.com/etri-crossmodal/gbswt5.git

How to load a model

import gbswt5
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("etri-lirs/gbst-kebyt5-base-preview")
model = AutoModelForSeq2SeqLM.from_pretrained("etri-lirs/gbst-kebyt5-base-preview")

Dependency

pytorch>=1.8.0
transformers>=4.27.0
einops>=0.6.0

Citation

Written in korean only.

@article{shin2023tflm,
  title={Towards Korean-centric Token-free Pretrained Language Model},
  author={Shin, Hur and Ryu, Lee and Seo, Seong and Lim.},
  journal={Proceedings of the 35th Annual Conference on Human and Cognitive Language Technology, pp. 711-715.},
  year={2023}
}

Acknowledgement

This software was supported by the Institute of Information & communication Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT). (No. RS-2022-00187238, Development of Large Korean Language Model Technology for Efficient Pre-training)
This software includes lucidrains/charformer-pytorch GitHub project for GBST implementation, which distributed under MIT License. Copyright (c) 2021 Phil Wang. all rights reserved. (Original Code URL: https://github.com/lucidrains/charformer-pytorch)
This software includes HuggingFace transformers's T5 implementation for GBST-enabled T5 model, which distributed under Apache 2.0 License. Copyright 2018- The Huggingface team. All rights reserved.

We are grateful for their excellent works.

Recommend Projects