mmBERT: A Modern Multilingual Encoder with Annealed Language Learning

2025-09-08

AI Summary

To address the poor pretraining efficacy and weak cross-lingual understanding of multilingual encoders—particularly for low-resource languages—this paper proposes an efficient encoder-only multilingual model. Methodologically, it introduces two novel training strategies: inverse masking ratio scheduling and inverse temperature sampling, which dynamically incorporate over 1,700 low-resource languages during the training decay phase. Leveraging a 3-trillion-token multilingual corpus, the model is pretrained via dynamic data scheduling integrated with curriculum learning. Empirically, it achieves state-of-the-art performance across classification and retrieval tasks, matching or exceeding the capabilities of o3 and Gemini 2.5 Pro on both high- and low-resource languages. The model demonstrates significantly enhanced multilingual generalization and markedly improved representation quality for low-resource languages, establishing new benchmarks in efficient multilingual encoding.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Encoder-only languages models are frequently used for a variety of standard machine learning tasks, including classification and retrieval. However, there has been a lack of recent research for encoder models, especially with respect to multilingual models. We introduce mmBERT, an encoder-only language model pretrained on 3T tokens of multilingual text in over 1800 languages. To build mmBERT we introduce several novel elements, including an inverse mask ratio schedule and an inverse temperature sampling ratio. We add over 1700 low-resource languages to the data mix only during the decay phase, showing that it boosts performance dramatically and maximizes the gains from the relatively small amount of training data. Despite only including these low-resource languages in the short decay phase we achieve similar classification performance to models like OpenAI's o3 and Google's Gemini 2.5 Pro. Overall, we show that mmBERT significantly outperforms the previous generation of models on classification and retrieval tasks -- on both high and low-resource languages.

Problem

• Addresses lack of recent multilingual encoder research
• Introduces mmBERT for 1800+ language pretraining
• Improves performance on low-resource language tasks

Innovation

• Annealed language learning with inverse mask ratio
• Inverse temperature sampling for multilingual training
• Late low-resource language addition during decay phase