Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment

2025-10-06

AI Summary

RLHF and DPO training suffer simultaneously from preference data contamination, reward over-optimization, and length bias—yet existing methods address at most one issue or rely on multiple reward models, lacking theoretical generalization guarantees. Method: We propose COV, a unified algorithmic framework that jointly mitigates all three deficiencies without explicit reward modeling or multi-reward estimation. COV incorporates length regularization and bias correction, supporting both offline and online training. Contribution/Results: We theoretically prove the equivalence between DPO-COV and RLHF-COV, and establish a generalization error bound with explicit length regularization, achieving optimal convergence rates under clean data assumptions. Empirically, COV significantly improves alignment quality, suppresses verbosity and over-optimization, and maintains strong robustness under contaminated preference data.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) are important techniques to align large language models (LLM) with human preference. However, the quality of RLHF and DPO training is seriously compromised by extit{ extbf{C}orrupted} preference, reward extit{ extbf{O}veroptimization}, and bias towards extit{ extbf{V}erbosity}. To our knowledge, most existing works tackle only one of these important issues, and the few other works require much computation to estimate multiple reward models and lack theoretical guarantee of generalization ability. In this work, we propose RLHF- extbf{COV} and DPO- extbf{COV} algorithms that can simultaneously mitigate these three issues, in both offline and online settings. This ability is theoretically demonstrated by obtaining length-regularized generalization error rates for our DPO-COV algorithms trained on corrupted data, which match the best-known rates for simpler cases with clean data and without length regularization. Moreover, our DPO-COV algorithm is simple to implement without reward estimation, and is proved to be equivalent to our RLHF-COV algorithm, which directly implies the equivalence between the vanilla RLHF and DPO algorithms. Experiments demonstrate the effectiveness of our DPO-COV algorithms under both offline and online settings.

Problem

• Mitigating corrupted preference in RLHF/DPO alignment
• Addressing reward overoptimization in language model training
• Reducing verbosity bias in offline and online settings

Innovation

• Simultaneously mitigates corruption, overoptimization, and verbosity
• Uses length-regularized generalization with theoretical guarantees
• Simple implementation without reward estimation in DPO-COV