Pretraining Finnish ModernBERTs

2025-11-12

AI Summary

To address the limited long-context modeling capability of Finnish and related languages under multilingual resource constraints, this paper pretrains six ModernBERT encoder models of varying scales. Methodologically, we extend the BERT architecture and perform self-supervised pretraining on large-scale Finnish and closely related language corpora, with explicit optimization for sequences exceeding 512 tokens; we further conduct a systematic analysis of end-stage data curation strategies. Our contributions are threefold: (1) the first dedicated, scalable multilingual BERT family specifically designed for the Finno-Ugric language family; (2) consistent and significant performance gains over general-purpose multilingual models (e.g., mBERT, XLM-R) and the monolingual FinBERT on long-text NLP tasks—including question answering and document classification; and (3) full open-sourcing of all models and training code to facilitate research and applications in low-resource, long-context NLP.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

This paper reports on pretraining ModernBERT encoder models in six different sizes, ranging from 51M to 475M parameters, with a focus on limited multilingualism, emphasizing languages relevant to Finland. Our models are competitive with, or superior to, existing multilingual models. They outperform monolingual models on tasks that require a context longer than 512 tokens. We present empirical results on using different data in the final stage of training. The code and models are publicly released.

Problem

• Pretraining Finnish-focused multilingual BERT models
• Developing competitive models with limited multilingual data
• Enhancing performance on long-context language tasks

Innovation

• Pretrained Finnish ModernBERT encoder models
• Models range from 51M to 475M parameters
• Focus on limited multilingualism for Finland