Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation

2025-12-11

AI Summary

This paper addresses the challenge of adapting large language models to mid- and low-resource languages. Instead of conventional continued pretraining, it proposes an efficient multilingual expansion paradigm based on model upscaling. Its three core contributions are: (1) the first systematic empirical validation that model scaling outperforms data augmentation in improving low-resource language performance; (2) a scalable adaptation strategy that significantly reduces target-language data requirements—by multiple-fold for smaller models under continued training—while effectively mitigating catastrophic forgetting of source-language (e.g., English) capabilities; and (3) an investigation of language-level parameter merging (e.g., PMT, TIES-Merging), demonstrating that merging upsampled models yields superior multilingual performance and consistency compared to merging smaller base models, thereby enabling modular, high-fidelity multilingual systems. Experiments include FLOP-matched ablations, comprehensive multilingual capability evaluation, and forgetting analysis.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Achieving high-performing language models which include medium- and lower-resource languages remains a challenge. Massively multilingual models still underperform compared to language-specific adaptations, especially at smaller model scales. In this work, we investigate scaling as an efficient strategy for adapting pretrained models to new target languages. Through comprehensive scaling ablations with approximately FLOP-matched models, we test whether upscaling an English base model enables more effective and resource-efficient adaptation than standard continued pretraining. We find that, once exposed to sufficient target-language data, larger upscaled models can match or surpass the performance of smaller models continually pretrained on much more data, demonstrating the benefits of scaling for data efficiency. Scaling also helps preserve the base model's capabilities in English, thus reducing catastrophic forgetting. Finally, we explore whether such scaled, language-specific models can be merged to construct modular and flexible multilingual systems. We find that while merging remains less effective than joint multilingual training, upscaled merges perform better than smaller ones. We observe large performance differences across merging methods, suggesting potential for improvement through merging approaches specialized for language-level integration.

Problem

• Scaling pretrained models for efficient adaptation to new languages
• Reducing catastrophic forgetting while preserving base language capabilities
• Exploring merging strategies for modular multilingual system construction

Innovation

• Upscaling base models for efficient language adaptation
• Scaling reduces catastrophic forgetting in target languages
• Merging scaled models for modular multilingual systems