Distributionally Robust Listwise Preference Optimization

2026-07-02

AI Summary

This work addresses the uncertainty in listwise preference ranking for language model alignment—arising from inconsistent annotations, near-ties, or reward noise—by proposing a robust Plackett–Luce objective grounded in total variation distributionally robust optimization. The method directly optimizes ranking labels given a prompt and a candidate list, and reduces the worst-case enumeration complexity of rankings from \(K!\) to \(O(K \log K)\), thereby providing the first theoretical guarantees for both offline and online listwise preference learning. Experiments demonstrate that the approach preserves performance on clean labels in offline settings while significantly improving robustness to noise; in online settings, it enhances the reliability of candidate expansion. Consistent gains are observed across both reward models and GPT-4-based evaluation metrics.

Abstract

Existing robust preference optimization for language-model alignment mainly studies pairwise supervision and places robustness at the dataset, prompt, or preference-pair level. We instead study listwise preference optimization under ranking-label uncertainty: given a prompt and a candidate list, the observed ranking over that list may be ambiguous due to annotator inconsistency, near-ties, lossy rankwise feedback, or reward-model noise. We propose a pointwise total-variation robust Plackett--Luce objective that directly robustifies the ranking label conditional on the candidate list. The robust loss admits an exact decomposition into the nominal PL loss plus a worst-case PL correction, and the worst-case ranking is obtained by sorting current implicit scores in ascending order, reducing the inner maximization from $K!$ enumeration to $O(K\log K)$. This tractable structure yields strong offline and online optimization guarantees. In the offline fixed-list setting, the robust objective is convex and projected stochastic subgradient reaches global $ε$-suboptimality with $O(ε^{-2})$ sample complexity. In the online policy-induced setting, where candidate lists are generated by the current policy, we establish weak convexity and $\widetilde O(ε^{-2})$ Moreau-envelope stationarity. Experiments in offline LLM alignment show that the proposed robust correction largely preserves performance under clean labels and improves robustness under noise. In online alignment, it makes reward-model-ranked candidate expansion more reliable and improves both reward-model and external GPT-4 judge metrics.

Problem

• distributionally robust optimization
• listwise preference
• ranking-label uncertainty
• language model alignment
• Plackett-Luce model

Innovation

• distributionally robust optimization
• listwise preference learning
• Plackett-Luce model
• ranking uncertainty
• total variation robustness