My NER Experiments with ModernBERT and Ettin on the official CoNLL-2003 NER dataset.

Current results so far - at the moment with only one run with a specific configuration (= batch size, learning rate, nunber of epochs, context size).

All experiments are performed with latest Flair version, Micro F1-Score on the development set:

Performance is currently very low - I opened an issue about this here.

After some debugging, it seems that the original tokenizer prepends a whitespace when performing the tokenizer() call in combination with the is_split_into_words option (which is needed for token classification tasks).

One quick workaround was tested: using the RobertaTokenizerFast with add_prefix_space=True. For this Flair needs to be slighly modified:

diff --git a/flair/embeddings/transformer.py b/flair/embeddings/transformer.py
index fdb16eea2..09877a766 100644
--- a/flair/embeddings/transformer.py
+++ b/flair/embeddings/transformer.py
@@ -1080,9 +1080,17 @@ class TransformerEmbeddings(TransformerBaseEmbeddings):
 
         if tokenizer_data is None:
             # load tokenizer and transformer model
-            self.tokenizer = AutoTokenizer.from_pretrained(
-                model, add_prefix_space=True, **transformers_tokenizer_kwargs, **kwargs
-            )
+            if "modernbert" in model.lower():
+                from transformers import RobertaTokenizerFast
+                print("ModernBERT detected, using RoBERTa tokenizer!")
+                self.tokenizer = RobertaTokenizerFast.from_pretrained(
+                    model, add_prefix_space=True, **transformers_tokenizer_kwargs, **kwargs
+                )
+                self.tokenizer.model_max_length = 512
+            else:
+                self.tokenizer = AutoTokenizer.from_pretrained(
+                    model, add_prefix_space=True, **transformers_tokenizer_kwargs, **kwargs
+                )
             try:
                 self.feature_extractor = AutoFeatureExtractor.from_pretrained(model, apply_ocr=False, **kwargs)
             except OSError:

After this fix we get better performance:

More runs:

Please watch and star this repo for updates.

I fixed the Tokenizer issue on the model side by forking the original ModernBERT Large model and applying the fixes in the Tokenizer config. This model is available here under the stefan-it/ModernBERT-large-tokenizer-fix Model Hub identifier. Using this new model has the huge advantage, that fixes in the Flair library are no longer needed!

Additionally, new experiments are conducted by pooling the first and last subtoken (instead of only using the first subtoken embedding), as this improved performance.

Here are new runs, with latest Flair and Transformers version:

Results on the test set (also Micro F1-Score):

The same hyper-parameter search is used for the recently released Ettin series. Ettin uses the same tokenizer as ModernBERT, so the exact same tokenizer fix is applied - the forked model can be found here

The first_last pooling strategy is also used - here are the results:

Results on the test set (Micro F1-Score):