The Crucial Role of Samplers in Online Direct Preference Optimization

2024-09-29

arXiv.org

AI Summary

This work investigates the impact of sampling strategies on the convergence of online Direct Preference Optimization (DPO). Addressing the limitation of standard uniform sampling—which yields only linear convergence—we provide the first theoretical proof that the choice of sampler fundamentally determines the convergence order of DPO, and propose the first online sampler with rigorous quadratic convergence guarantees. Our method integrates Bayesian posterior distribution modeling with a logit-weighted mixture mechanism, ensuring both theoretical soundness and practical robustness. On the Safe-RLHF benchmark, it achieves over 7.4% improvement in preference alignment accuracy compared to standard DPO. Beyond establishing a formal link between sampling design and convergence rate, this work pioneers a new paradigm wherein deliberate sampler construction drives substantial performance gains in preference optimization algorithms.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

Direct Preference Optimization (DPO) has emerged as a stable, scalable, and efficient solution for language model alignment. Despite its empirical success, the optimization properties, particularly the impact of samplers on its convergence rates, remain under-explored. In this paper, we provide a rigorous analysis of DPO's convergence rates with different sampling strategies under the exact gradient setting, revealing a surprising separation: uniform sampling achieves $ extbf{linear}$ convergence, while our proposed online sampler achieves $ extbf{quadratic}$ convergence. We further adapt the sampler to practical settings by incorporating posterior distributions and logit mixing, demonstrating improvements over previous methods. For example, it outperforms vanilla DPO by over $7.4$% on Safe-RLHF dataset. Our results not only offer insights into the theoretical understanding of DPO but also pave the way for further algorithm designs.

Problem

• Direct Preference Optimization
• Convergence Speed
• Practical Performance

Innovation

• Direct Preference Optimization
• Convergence Speed
• Enhanced Sampling Strategy