mkmoisio / sts-en-to-fi-distillation

Overview

Semantic textual similarity (STS) is a metric measuring similarity of meanings of natural language text pieces. Majority of recent NLP research has focused on a small set of largest Indo-European languages and Chinese, and, languages with smaller speaker population, Finnish for instance, often lack annotated data required to train complex models.

This work experiments using knowledge distillation to transfer STS properties learnt from English into a model pre-trained on Finnish while bypassing the lack of annotated Finnish data. Further, we experiment distillation with different types of data, English-Finnish bilingual, English monolingual and random pseudo samples, to observe which properties of training data are really necessary.

We briefly follow the knowledge distillation approach described by Reimers at al. We exploit a good-known English trained Sentence-BERT fine-tuned for STS as a teacher f_teacher to train selected student models f_student so that f_teacher(s) ≈ f_student(s) and f_student(s) ≈ f_student (t) where s is a sentence in the source language and t is a sentence in target language.

Models

Model	Role	Origin
stsb-distilbert-base	Teacher + Baseline	sbert.net
bert-base-finnish-cased-v1 (FinBERT)	Baseline	TurkuNLP
xlm-roberta-base	Student + Baseline	🤗

Data

Set	Type	Size
Training, cross-lingual	Parallel bilingual	216,104
Training, inter-lingual	Unilingual	208,054
Training, extraction	Sampled	200,000
Development	Parallel bilingual	1,141
Test	Annotated bilingual	250

Training

Evaluation

Ref

This work mostly builds on the following:

• Reimers et al.
  • Sentence-BERT (repository)
  • Multilingual Knowledge Distillation
• Krishna et al.
  • Model Extraction
• Virtanen et al.

• FinBERT (repository)

• Tiedemann
  • Parallel data (OPUS)
• Cer et al.
  • Original test data