CM-Align: Consistency-based Multilingual Alignment for Large Language Models

2025-09-10

AI Summary

Current multilingual alignment of large language models (LLMs) is hindered by the poor quality of preference data: (i) using raw English responses as references introduces low-quality samples, and (ii) heuristic cross-lingual pairing yields preference pairs with high bias and noise. To address this, we propose a consistency-guided framework for multilingual preference construction. First, high-quality English references are selected based on semantic consistency among model responses. Then, robust cross-lingual preference pairs are generated via cross-lingual consistency evaluation and employed for Direct Preference Optimization (DPO). Our method achieves significant improvements over strong baselines across three mainstream LLMs and three representative multilingual tasks—including instruction following, translation, and reasoning. Results demonstrate that high-quality, consistency-driven preference data is critical for effective multilingual alignment. Moreover, our framework establishes a scalable, principled paradigm for constructing multilingual preference datasets, advancing the reliability and generalizability of alignment methods beyond English.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Current large language models (LLMs) generally show a significant performance gap in alignment between English and other languages. To bridge this gap, existing research typically leverages the model's responses in English as a reference to select the best/worst responses in other languages, which are then used for Direct Preference Optimization (DPO) training. However, we argue that there are two limitations in the current methods that result in noisy multilingual preference data and further limited alignment performance: 1) Not all English responses are of high quality, and using a response with low quality may mislead the alignment for other languages. 2) Current methods usually use biased or heuristic approaches to construct multilingual preference pairs. To address these limitations, we design a consistency-based data selection method to construct high-quality multilingual preference data for improving multilingual alignment (CM-Align). Specifically, our method includes two parts: consistency-guided English reference selection and cross-lingual consistency-based multilingual preference data construction. Experimental results on three LLMs and three common tasks demonstrate the effectiveness and superiority of our method, which further indicates the necessity of constructing high-quality preference data.

Problem

• Bridging performance gap in multilingual LLM alignment
• Addressing noisy multilingual preference data issues
• Improving cross-lingual consistency in preference optimization

Innovation

• Consistency-based multilingual data selection method
• Cross-lingual consistency for preference construction
• Quality-focused English reference selection approach