EuroBERT: Scaling Multilingual Encoders for European Languages

2025-03-07

AI Summary

With the rise of generative decoders, multilingual bidirectional encoders have become increasingly marginalized. Method: This paper introduces the EuroBERT family of multilingual encoders, designed to enhance coverage of major European and global languages and improve multitask capabilities. It systematically integrates long-context modeling (native 8,192-token sequences), dynamic curriculum learning, and multi-stage hybrid pretraining—incorporating high-quality European language data, mathematical texts, and code—into the RoBERTa architecture. Contribution/Results: EuroBERT achieves substantial gains in cross-lingual generalization and domain transfer, consistently outperforming mBERT, XLM-R, and multilingual LLaMA variants on multilingual understanding, mathematical reasoning, and code generation benchmarks. All models and the training framework are fully open-sourced, offering a renewed pathway for encoder-based paradigms in retrieval, classification, mathematical reasoning, and programming tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, we revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively supporting sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework.

Problem

• Develop multilingual encoders for European and global languages.
• Enhance performance in multilingual tasks, mathematics, and coding.
• Support long sequences up to 8,192 tokens efficiently.

Innovation

• Develops EuroBERT multilingual encoder models
• Supports sequences up to 8,192 tokens
• Publicly releases models and training framework