Towards Disentangled Preference Optimization Dynamics Beyond Likelihood Displacement

2026-04-20

AI Summary

This work addresses the prevalent likelihood shift problem in preference optimization, wherein suppressing dispreferred responses inadvertently weakens preferred ones, with no general mechanism to avoid it. The authors propose an incentive-score decomposition framework that reveals mainstream preference objectives share identical local update directions, differing only in weighting coefficients. Building on this insight, they introduce the first verifiable Decoupling Band (DB) condition to determine whether training avoids likelihood shift. Furthermore, they design a plug-and-play Reward Calibration (RC) method that adaptively satisfies the DB condition without altering the original objective. Experiments demonstrate that RC effectively steers optimization toward decoupled dynamics, substantially mitigating likelihood shift and consistently improving downstream task performance across diverse preference optimization objectives.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

Preference optimization is widely used to align large language models (LLMs) with human preferences. However, many margin-based objectives suppress the chosen response along with the rejected one, a phenomenon known as likelihood displacement, and no general mechanism currently prevents this across objectives. We bridge this gap by presenting a unified \emph{incentive-score decomposition} of preference optimization, revealing that diverse objectives share identical local update directions and differ only in their scalar weighting coefficients. Building on this decomposition, by analyzing the dynamics of the chosen/rejected likelihoods, we identify the \emph{disentanglement band} (DB), a simple, testable condition that characterizes when training can avoid likelihood displacement by realizing the preferred pathway: suppressing the loser while maintaining the winner, possibly after an initial transient. Leveraging the DB, we propose a plug-and-play \emph{reward calibration} (RC) that adaptively rebalances chosen versus rejected updates to satisfy the DB and mitigate likelihood displacement, without redesigning the base objective. Empirical results show that RC steers training toward more disentangled dynamics and often improves downstream performance across a range of objectives. Our code is available at https://github.com/IceyWuu/DisentangledPreferenceOptimization.

Problem

• preference optimization
• likelihood displacement
• large language models
• margin-based objectives
• disentangled dynamics

Innovation

• incentive-score decomposition
• disentanglement band
• likelihood displacement
• reward calibration
• preference optimization