LOLA - An Open-Source Massively Multilingual Large Language Model

2024-09-17

arXiv.org

AI Summary

To address the challenge of balancing linguistic diversity and computational efficiency in multilingual large language models (LLMs), this paper introduces LOLA, an open-source sparse Mixture-of-Experts (MoE) LLM supporting over 160 languages. Methodologically, we first discover that the expert routing mechanism inherently captures linguistic phylogeny—mitigating the “multilingual curse”—and further enhance it via curriculum-based multilingual data scheduling, language-aware tokenization, and routing regularization. Empirically, LOLA achieves state-of-the-art or near-state-of-the-art performance on multilingual understanding and generation benchmarks, significantly improving cross-lingual generalization and training scalability. All model weights, training configurations, and evaluation protocols are fully open-sourced to ensure reproducibility and facilitate efficient downstream adaptation.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

This paper presents LOLA, a massively multilingual large language model trained on more than 160 languages using a sparse Mixture-of-Experts Transformer architecture. Our architectural and implementation choices address the challenge of harnessing linguistic diversity while maintaining efficiency and avoiding the common pitfalls of multilinguality. Our analysis of the evaluation results shows competitive performance in natural language generation and understanding tasks. Additionally, we demonstrate how the learned expert-routing mechanism exploits implicit phylogenetic linguistic patterns to potentially alleviate the curse of multilinguality. We provide an in-depth look at the training process, an analysis of the datasets, and a balanced exploration of the model's strengths and limitations. As an open-source model, LOLA promotes reproducibility and serves as a robust foundation for future research. Our findings enable the development of compute-efficient multilingual models with strong, scalable performance across languages.

Problem

• Multilingual Language Model
• Open-Source
• Natural Language Processing

Innovation

• Multilingual Processing
• Open-Source Architecture
• Language Pattern Recognition