honghanhh / terminology-extraction

Terminology extraction on ACTER using Transformer-based language models

Terminology Extraction

We experiment with different Transformer-based pre-trained language models for the terminology extraction task, namely XLNet, BERT, DistilBERT, and RoBERTa, with additional techniques, including class weighting in order to reduce the significant imbalance in the training data, as well as rule-based term expansion and filtering. Our experiments are conducted on the ACTER dataset covering 3 languages and 3 domains. The results prove to be competitive on English and French, and the proposed approach outperforms the state of the art (SOTA) on Dutch.

1. ACTER

The dataset structure as well as the distributions of terms per domain per language are demonstrated in data.exploration.ipynb

---

2. Models and Architecture

2.1 Models

For each language, we examine several pretrained language models using SimpleTransformers as the following table.

Model	English dataset	French dataset	Dutch dataset
Multilingual BERT (uncased)	x	x	x
Multilingual BERT (cased)	x	x	x
Monolingual English BERT (uncased)	x
Monolingual English BERT (cased)	x
RoBERTa	x
DistilBERT (uncased)	x
DistilBERT (cased)	x
Multilingual DistilBERT (cased)			x
XLNet	x
CamemBERT		x

2.2 Architecture

The worflow of our implementation:

The code insides ./core_model/ is an example of how we implemented on French dataset on CamemBERT. Preferable to run on Google Collab to take advantage of GPU (in case your local machine does not support).

---

3. Results

All the saved prediction results of mentionned pretrained models on 3 languages are saved in folder ./results/weighted_results/.

---

4. References

• Shared Task on Automatic Term Extraction Using the Annotated Corpora for Term Extraction Research (ACTER) Dataset.
• TALN-LS2N System for Automatic Term Extraction.

---

Contributors:

• 🐮 TRAN Thi Hong Hanh 🐮
• Prof. Senja POLLAK
• Prof. Antoine DOUCET
• Prof. Matej MARTINC
• Prof. Andraž REPAR

terminology-extraction's Issues

Apply BERT on ACTER dataset

• Reimplement baseline of TermEval 2020's ATE
• Ref: TALN-LS2N System for Automatic Term Extraction
  • BERT and its invariants

Reproduce the scores

Hi, I want to thank you for sharing the code and formulate this task into sequence labeling.
As the result shows, it outperforms all the participants mentioned in the original paper.

I have cloned your code and tried to reproduce the scores you provided in the table, however, I could not reproduce such good performance. I only changed the model and model type to bert and bert-base-uncased. Then, add model arguments as you mentioned in Readme. Should I adjust anything that you didn't mention in the description?

Best.

Preprocessing test set

Hey Hahn,

Went through the code and found one thing worth checking :) . In preprocessing.py (in line 89) you remove the sentences that do not have terms for the train set, right? Do you do the same for the test set on which you apply the trained model? If so, I imagine the classifier would have an easier job of only predicting terms for sentences containing terms? Predictions for sentences that have no terms inside would on the other hand probably produce some false positives that would be added to the final list and decrease overall precision?

Analyze ACTER dataet

• Dataset: ACTER
  • Languages
  • Domains
  • Term Distributions/Frequencies/Ratio
    ...

Paper summary

Task 1: Read papers

• TermEval 2020: Shared Task on Automatic Term Extraction Using the
  Annotated Corpora for Term Extraction Research (ACTER) Dataset
• TermEval 2020: TALN-LS2N System for Automatic Term Extraction

Evaluation metrics