DPO-Shift: Shifting the Distribution of Direct Preference Optimization

2025-02-11

AI Summary

DPO and its variants are widely adopted for human preference alignment, yet suffer from “likelihood drift”: the generation probability of preferred responses monotonically decreases during training, undermining model stability and alignment quality. This work provides the first theoretical analysis quantifying the fundamental trade-off between increasing the likelihood of chosen responses and preserving the reward margin. Building on this insight, we propose a parameter-controllable probability distribution reweighting and gradient correction mechanism within the DPO framework, enabling targeted calibration of the preferred-response distribution. Our method incurs no additional model parameters, human annotations, or computational overhead, and seamlessly integrates with standard preference-based training pipelines. Empirical evaluation on MT-Bench and pairwise win-rate metrics demonstrates significant improvements over baseline DPO, effectively mitigating probability decay while enhancing generation consistency and robustness to preference alignment.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Humans and AI: Learning Human Values and Preferences
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Direct Preference Optimization (DPO) and its variants have become increasingly popular for aligning language models with human preferences. These methods aim to teach models to better distinguish between chosen (or preferred) and rejected (or dispreferred) responses. However, prior research has identified that the probability of chosen responses often decreases during training, and this phenomenon is known as likelihood displacement. To tackle this challenge, in this work we introduce method to controllably shift the distribution of the chosen probability. Then, we show that method exhibits a fundamental trade-off between improving the chosen probability and sacrificing the reward margin, as supported by both theoretical analysis and experimental validation. Furthermore, we demonstrate the superiority of method over DPO on downstream tasks such as MT-Bench and a designed win rate experiment. We believe this study shows that the likelihood displacement issue of DPO can be effectively mitigated with a simple, theoretically grounded solution. Our code is available at https://github.com/Meaquadddd/DPO-Shift.

Problem

• address likelihood displacement in DPO
• control distribution of chosen probability
• improve model alignment with human preferences

Innovation

• Controllable distribution shifting
• Mitigating likelihood displacement
• Trade-off improvement and sacrifice