EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

2024-09-26

arXiv.org

AI Summary

To address the insufficient coverage and weak performance of low-resource languages in multilingual models, this paper introduces EMMA-500—the first large-scale multilingual continual pretraining model supporting 546 languages. Methodologically, it builds upon the Llama 2-7B architecture and constructs MaLA, a high-quality, multi-domain multilingual dataset. It pioneers a massive-scale multilingual continual pretraining paradigm, augmented with cross-lingual vocabulary expansion and position encoding adaptation. The contributions are threefold: (1) substantial improvements in low-resource language understanding and generation, achieving an average +23.6% gain across multilingual benchmarks; (2) open release of the MaLA dataset, model weights, training scripts, and generated samples; and (3) advancement of reproducible research and practical deployment in multilingual NLP.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability. We release the MaLA corpus, EMMA-500 model weights, scripts, and model generations.

Problem

• Enhancing multilingual performance across 546 languages
• Improving language coverage for low-resource languages
• Expanding large language models' language capacity

Innovation

• Multilingual model EMMA-500
• MaLA corpus compilation
• Continual pre-training Llama 2