Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs

2025-02-18

AI Summary

This study addresses the critical shortage of AI support for low-resource Southeast Asian (SEA) languages. We introduce Sailor2, the first open-source multilingual large language model (LLM) series—comprising 1B, 8B, and 20B parameter variants—designed specifically for 13 SEA languages while retaining strong Chinese and English capabilities. Methodologically, we propose a novel SEA-centric continual pretraining paradigm over 500B tokens (400B SEA-specific data + 100B cross-lingual replay), built upon an optimized Qwen2.5 architecture. We further release a comprehensive five-stage multilingual LLM development handbook covering data curation, training, evaluation, and more. Evaluation shows Sailor2-20B achieves parity with GPT-4o on SEA-language benchmarks (50–50 win rate). All models and full reproducible training pipelines are released under the Apache 2.0 license, substantially advancing equitable AI ecosystem development in the region.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Sailor2 is a family of cutting-edge multilingual language models for South-East Asian (SEA) languages, available in 1B, 8B, and 20B sizes to suit diverse applications. Building on Qwen2.5, Sailor2 undergoes continuous pre-training on 500B tokens (400B SEA-specific and 100B replay tokens) to support 13 SEA languages while retaining proficiency in Chinese and English. Sailor2-20B model achieves a 50-50 win rate against GPT-4o across SEA languages. We also deliver a comprehensive cookbook on how to develop the multilingual model in an efficient manner, including five key aspects: data curation, pre-training, post-training, model customization and evaluation. We hope that Sailor2 model (Apache 2.0 license) will drive language development in the SEA region, and Sailor2 cookbook will inspire researchers to build more inclusive LLMs for other under-served languages.

Problem

• Develops multilingual models for SEA languages
• Enhances language proficiency in SEA region
• Provides methodology for inclusive LLM development

Innovation

• Multilingual language models
• Continuous pre-training
• Comprehensive cookbook development