moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT

2026-06-21

AI Summary

This work addresses the lack of high-performance encoder models supporting Portuguese and long-context processing, which hinders performance in tasks such as information retrieval and text classification. We present the first adaptation of the ModernBERT architecture to Portuguese, leveraging continued pretraining on 60 billion tokens of high-quality text from FineWeb2, filtered for educational and STEM content. Our model, moBERTo, incorporates a dedicated tokenizer, subword-matching embedding transfer, and long-context post-training strategies. It integrates rotary positional embeddings, alternating local-global attention, Flash Attention, and unpadding techniques to maintain strong long-sequence modeling capabilities while significantly boosting efficiency and effectiveness. moBERTo achieves state-of-the-art average reranking nDCG@10 across three Portuguese retrieval benchmarks and outperforms all competitors on the PLUE-PT evaluation suite, demonstrating the superiority of continued pretraining over training from scratch.

Abstract

Encoder-only transformer models remain essential for production NLP pipelines. We introduce moBERTo, a Portuguese adaptation of ModernBERT obtained through continued pretraining of the ModernBERT-base checkpoint on 60 billion tokens (5 epochs over a 12-billion-token corpus curated from FineWeb2 and filtered with educational and STEM classifiers). We preserve the original architecture, including rotary positional embeddings, alternating local-global attention, flash attention, and unpadding. We evaluate moBERTo across information retrieval (including long-context retrieval at up to 8,192 tokens), document classification, named entity recognition, and natural language understanding. Our best variant, which combines a Portuguese tokenizer with subword-matching embedding transfer and long-context post-training, achieves the highest average reranking nDCG@10 across three Portuguese retrieval benchmarks and the best results on PLUE-PT. Through ablation studies, we show that (i) continued pretraining is strongly preferable to training from scratch, particularly for preserving long-context capabilities; (ii) tokenizer adaptation improves token-level tasks but degrades long-context retrieval; (iii) a dedicated long-context post-training phase at 8,192 tokens further improves reranking and NER; and (iv) encoder-only architectures remain competitive with larger decoder-only alternatives for discriminative tasks. We publicly release the model weights at https://huggingface.co/Tropic-AI/moBERTo and training data at https://huggingface.co/datasets/Tropic-AI/moberto-pretraining-dataset-c4-compatible on Hugging Face.

Problem

• Portuguese NLP
• encoder-only transformer
• continued pretraining
• long-context retrieval
• language model adaptation

Innovation

• continued pretraining
• long-context retrieval
• encoder-only transformer
• tokenizer adaptation
• Portuguese NLP