Language Chain in Alignment: Cross-Lingual Ranking Preference Optimization

2026-08-24

AI Summary

本文针对大型语言模型在非英语语言中性能不佳的问题，提出了一种利用英语言偏好知识优化目标语言偏好的跨语言排序偏好优化框架CRPO。

Abstract

The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-Lingual Ranking Preference Optimization (CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment.

Problem

• Cross-Lingual
• Ranking Preference
• Language Models
• Alignment
• Multilingual

Innovation

• Cross-Lingual Ranking Preference Optimization
• Hierarchical Structure
• LambdaLoss
• Preference Alignment
• Multilingual Setup