AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment

2025-09-29

AI Summary

Multilingual large language models (LLMs) suffer from substantial performance degradation on non-dominant languages and exhibit insufficient cross-lingual representation alignment, hindering effective knowledge transfer. To address this, we propose AlignX—a two-stage representation-level alignment framework. In the first stage, fine-grained multilingual semantic alignment is achieved via contrastive learning; in the second stage, language-specific features are integrated with multilingual instruction tuning to jointly optimize cross-lingual understanding and generation capabilities. Unlike conventional output-layer alignment or single-stage fine-tuning, AlignX jointly models semantic commonalities and linguistic idiosyncrasies at the representation level. Extensive experiments across 12 languages and multiple pretrained LLMs (e.g., mBERT, XGLM) demonstrate that AlignX significantly narrows the multilingual performance gap: it yields average improvements of 4.2–7.8 percentage points on cross-lingual understanding benchmarks (XNLI, XCOPA) and the XGen multilingual generation task, while markedly enhancing representation alignment quality and generalization capacity.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multilingual large language models (LLMs) possess impressive multilingual understanding and generation capabilities. However, their performance and cross-lingual alignment often lag for non-dominant languages. A common solution is to fine-tune LLMs on large-scale and more balanced multilingual corpus, but such approaches often lead to imprecise alignment and suboptimal knowledge transfer, struggling with limited improvements across languages. In this paper, we propose AlignX to bridge the multilingual performance gap, which is a two-stage representation-level framework for enhancing multilingual performance of pre-trained LLMs. In the first stage, we align multilingual representations with multilingual semantic alignment and language feature integration. In the second stage, we stimulate the multilingual capability of LLMs via multilingual instruction fine-tuning. Experimental results on several pre-trained LLMs demonstrate that our approach enhances LLMs' multilingual general and cross-lingual generation capability. Further analysis indicates that AlignX brings the multilingual representations closer and improves the cross-lingual alignment.

Problem

• Improving multilingual performance for non-dominant languages
• Addressing imprecise alignment in multilingual representation transfer
• Enhancing cross-lingual generation capabilities through semantic alignment

Innovation

• Two-stage framework aligns multilingual representations semantically
• Integrates language features to enhance cross-lingual alignment
• Uses multilingual instruction fine-tuning to stimulate capabilities