English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training

2026-04-14

AI Summary

This study addresses the imbalance in multilingual performance of large language models caused by post-training’s heavy reliance on English. Through 220 controlled fine-tuning experiments, it systematically investigates the interplay among language coverage, model scale (up to 8B parameters), and task domains—specifically mathematical reasoning and API calling—using parallel-translated multilingual data and supervised fine-tuning. The findings reveal that incorporating even a single non-English language enhances both English performance and cross-lingual generalization. Zero-shot transfer under high linguistic diversity matches the performance of low-diversity settings where the target language is explicitly included. Broadening language coverage consistently improves performance across all languages, with low-resource languages showing substantial gains and high-resource languages approaching saturation without degradation. These results demonstrate that English-only post-training is suboptimal and underscore the effectiveness and necessity of multilingual post-training.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Despite the widespread multilingual deployment of large language models, post-training pipelines remain predominantly English-centric, contributing to performance disparities across languages. We present a systematic, controlled study of the interplay between training language coverage, model scale, and task domain, based on 220 supervised fine-tuning runs on parallel translated multilingual data mixtures spanning mathematical reasoning and API calling tasks, with models up to 8B parameters. We find that increasing language coverage during post-training is largely beneficial across tasks and model scales, with low-resource languages benefiting the most and high-resource languages plateauing rather than degrading. Even minimal multilinguality helps: incorporating a single non-English language improves both English performance and cross-lingual generalization, making English-only post-training largely suboptimal. Moreover, at sufficient language diversity, zero-shot cross-lingual transfer can match or exceed the effects of direct language inclusion in a low-diversity setting, although gains remain limited for typologically distant, low-resource languages.

Problem

• multilinguality
• post-training
• language disparity
• large language models
• cross-lingual transfer

Innovation

• multilingual post-training
• cross-lingual transfer
• language coverage
• supervised fine-tuning
• low-resource languages