Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages

2025-02-15

AI Summary

Extremely low-resource languages lack high-quality text generation models, as mainstream large language models (e.g., LLaMA, Qwen) support far fewer languages than multilingual encoders like XLM-R, leaving many languages without viable generative capabilities. Method: We propose an encoder-decoder weight-sharing framework that leverages XLM-R’s pretrained multilingual semantic representations, eliminating the need for separate decoder pretraining for extremely low-resource languages. Our approach enables cross-lingual semantic space transfer and lightweight decoder adaptation. Contribution/Results: We empirically validate the framework on four Chinese minority languages—the first such demonstration for these languages. Experiments show that our method significantly outperforms baseline models with several times more parameters across multiple downstream generation tasks, establishing the first efficient and practical text generation solution for languages lacking existing LLM support.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While multilingual language models like XLM-R have advanced multilingualism in NLP, they still perform poorly in extremely low-resource languages. This situation is exacerbated by the fact that modern LLMs such as LLaMA and Qwen support far fewer languages than XLM-R, making text generation models non-existent for many languages in the world. To tackle this challenge, we propose a novel framework for adapting multilingual encoders to text generation in extremely low-resource languages. By reusing the weights between the encoder and the decoder, our framework allows the model to leverage the learned semantic space of the encoder, enabling efficient learning and effective generalization in low-resource languages. Applying this framework to four Chinese minority languages, we present XLM-SWCM, and demonstrate its superior performance on various downstream tasks even when compared with much larger models.

Problem

• Improves multilingual models for low-resource languages.
• Enables text generation in extremely low-resource languages.
• Reuses encoder weights for efficient language learning.

Innovation

• Shared weights pretraining method
• Encoder-decoder weight reuse
• XLM-SWCM for low-resource languages