Toward LLMs Beyond English-Centric Development

2026-05-15

AI Summary

Current large language models exhibit a pronounced English-centric bias, limiting their effectiveness in supporting non-English languages and cultural understanding. This study systematically evaluates the cost-effectiveness of continued pretraining for target-language adaptation by analyzing generation outputs from open-source large language models and conducting cross-lingual cultural comprehension assessments. The findings reveal that leveraging English-centric resources for continued pretraining offers no significant cost advantage in enhancing non-English performance, thereby underscoring the necessity of dedicated, language-specific investment. This work provides empirical evidence and strategic guidance for multilingual large language model development, advocating for a more balanced and linguistically tailored training paradigm.

Abstract

Through an analysis of sequences generated by open-weight large language models (LLMs), we demonstrate that LLMs are heavily biased toward English. While continual pre-training is commonly used to adapt LLMs to a target language, we show that it does not offer a cost advantage over training from scratch, even for improving cultural understanding in the target language. These findings suggest that dedicated per-language investment may become increasingly important for future LLM development, rather than relying primarily on the expansion of English-centric resources.

Problem

• English-centric bias
• large language models
• multilingual adaptation
• cultural understanding
• language-specific development

Innovation

• English-centric bias
• continual pre-training
• multilingual LLMs
• language-specific investment
• cultural understanding