• GERTuraX is a series of pretrained encoder-only language models for German.

• The models are ELECTRA-based and pretrained with the TEAMS approach on the CulturaX corpus.

• In total, three different models were trained and released with pretraining corpus sizes ranging from 147GB to 1.1TB.

This repository hosts all necessary code to conduct the GERTuraX fine-tuning experiments on various downstream tasks using the awesome Flair library.

• 18.07.2025: Add new results for BarNER dataset.
• 12.07.2025: Add new results for German CoNLL-2003 Original.
• 20.06.2025: Add new results for ModernGBERT and GeistBERT.
• 09.02.2025: Add initial version of this repository.

First, Flair and other dependencies must be installed:

$ pip3 install -r requirements.txt

The following environment variables can be set:

Here's an example for the used JSON-based configuration format:

{
    "batch_sizes": [
        32,
        16
    ],
    "learning_rates": [
        1e-05,
        2e-05,
        3e-05,
        4e-05
    ],
    "epochs": [
        20
    ],
    "context_sizes": [
        0
    ],
    "seeds": [
        1,
        2,
        3,
        4,
        5
    ],
    "layers": "-1",
    "subword_poolings": [
        "first"
    ],
    "use_crf": false,
    "use_tensorboard": true,
    "hf_model": "deepset/gbert-base",
    "model_short_name": "gbert_base",
    "task": "ner/germeval14",
    "cuda": "0"
}

Hyper-parameter searches are possible, e.g. different batch sizes, learning rates, epochs or seeds can be set. The CUDA device id can be set via cuda (expecting the id as string).

After environment variables are set, the fine-tuning can be started with:

$ python3 script.py

The flair-log-parser.py script can be used to get an overview of best configurations and their correspondig F1-Scores.

GERTuraX and other German Language Models were fine-tuned on GermEval 2014 (NER), GermEval 2018 (Sentiment analysis), CoNLL-2003 (NER) and BarNER (NER).

We use the same hyper-parameters for GermEval 2014, GermEval 2018, CoNLL-2003 and BarNER as used in the GeBERTa paper (cf. Table 5) using 5 runs with different seed and report the averaged score, conducted with the awesome Flair library.

GERTuraX is the outcome of the last 12 months of working with TPUs from the awesome TRC program and the TensorFlow Model Garden library.

Many thanks for providing TPUs!

Made from Bavarian Oberland with ❤️ and 🥨.