Semantic textual similarity (STS) is a metric measuring similarity of meanings of natural language text pieces. Majority of recent NLP research has focused on a small set of largest Indo-European languages and Chinese, and, languages with smaller speaker population, Finnish for instance, often lack annotated data required to train complex models.
This work experiments using knowledge distillation to transfer STS properties learnt from English into a model pre-trained on Finnish while bypassing the lack of annotated Finnish data. Further, we experiment distillation with different types of data, English-Finnish bilingual, English monolingual and random pseudo samples, to observe which properties of training data are really necessary.
We briefly follow the knowledge distillation approach described by Reimers at al. We exploit a good-known English trained Sentence-BERT fine-tuned for STS as a teacher fteacher to train selected student models fstudent so that fteacher(s) โ fstudent(s) and fstudent(s) โ fstudent (t) where s is a sentence in the source language and t is a sentence in target language.
Model | Role | Origin |
---|---|---|
stsb-distilbert-base | Teacher + Baseline | sbert.net |
bert-base-finnish-cased-v1 (FinBERT) | Baseline | TurkuNLP |
xlm-roberta-base | Student + Baseline | ๐ค |
Set | Type | Size |
---|---|---|
Training, cross-lingual | Parallel bilingual | 216,104 |
Training, inter-lingual | Unilingual | 208,054 |
Training, extraction | Sampled | 200,000 |
Development | Parallel bilingual | 1,141 |
Test | Annotated bilingual | 250 |
This work mostly builds on the following:
- Reimers et al.
- Krishna et al.
- Virtanen et al.
- Tiedemann
- Cer et al.