Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph

2026-05-08

AI Summary

Existing pairwise preference optimization methods struggle to effectively leverage the rich preference structures inherent in multi-response scenarios, often neglecting preference transitivity and suffering from training instability. This work proposes Graph Direct Preference Optimization (GraphDPO), which for the first time models multi-response preferences as a directed acyclic graph and employs a graph-structured Plackett–Luce objective to aggregate neighborhood supervision signals. This approach explicitly captures preference transitivity while maintaining linear computational complexity. GraphDPO introduces equivalence classes to handle discrete preferences and supports answer anchoring with annealed scheduling, thereby unifying and generalizing the standard DPO framework. Experimental results demonstrate that GraphDPO significantly outperforms current pairwise and listwise alignment methods on reasoning and program synthesis tasks, validating the scalability and robustness of graph-based preference modeling.

Abstract

Direct Preference Optimization (DPO) aligns language models using pairwise preference comparisons, offering a simple and effective alternative to Reinforcement Learning (RL) from human feedback. However, in many practical settings, training data consists of multiple rollouts per prompt, inducing rich preference structure that pairwise DPO fails to exploit. Collapsing such data into independent pairs discards transitivity, introduces redundant or conflicting supervision, and can lead to unstable optimization. We propose Graph Direct Preference Optimization (GraphDPO), a principled generalization of DPO that operates over directed acyclic preference graphs induced by rollout rankings. GraphDPO encodes dominance relations as edges and optimizes a graph-structured Plackett--Luce-inspired objective that aggregates supervision over graph neighborhoods, enforcing transitivity while recovering standard DPO as a special case. To handle discrete or sparse signals, we introduce an equivalence-class construction where responses with identical preferences form graph layers, and intra-layer edges contribute zero loss, preventing spurious gradients. Despite leveraging full graph structure, GraphDPO maintains linear per-prompt complexity via efficient log-sum-exp aggregation. We further incorporate optional ground-truth anchoring by inserting verified solutions as dominant nodes and applying an annealed schedule that stabilizes early training while gradually relaxing oracle supervision. Experiments on reasoning and program synthesis tasks demonstrate superior performance, suggesting that graph-structured preference modeling is a scalable and robust alternative to pairwise and listwise alignment objectives.

Problem

• preference optimization
• language model alignment
• preference graph
• transitivity
• multi-rollout supervision

Innovation

• GraphDPO
• preference graph
• transitivity
• Plackett-Luce
• direct preference optimization