Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF

2024-05-29

arXiv.org

AI Summary

This work addresses the challenge of uncertainty estimation in reward modeling for Reinforcement Learning from Human Feedback (RLHF). We propose a unified framework that jointly handles online and offline preference data. Our core method introduces sign-modulated value-function regularization, enabling scalable integration of optimistic/pessimistic uncertainty principles into large language model RLHF pipelines, thereby achieving implicit joint optimization of reward and policy. Built upon maximum-likelihood reward estimation and implicit reward modeling, our approach avoids explicit reward modeling or sampling, ensuring full compatibility with standard RLHF paradigms. Theoretical analysis shows its convergence rate matches that of standard RL. Empirical evaluation on text summarization and dialogue tasks demonstrates substantial improvements over baselines, with enhanced stability and generalization across both online and offline settings.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Humans and AI: Learning Human Values and Preferences
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement learning from human feedback (RLHF) has demonstrated great promise in aligning large language models (LLMs) with human preference. Depending on the availability of preference data, both online and offline RLHF are active areas of investigation. A key bottleneck is understanding how to incorporate uncertainty estimation in the reward function learned from the preference data for RLHF, regardless of how the preference data is collected. While the principles of optimism or pessimism under uncertainty are well-established in standard reinforcement learning (RL), a practically-implementable and theoretically-grounded form amenable to large language models is not yet available, as standard techniques for constructing confidence intervals become intractable under arbitrary policy parameterizations. In this paper, we introduce a unified approach to online and offline RLHF -- value-incentivized preference optimization (VPO) -- which regularizes the maximum-likelihood estimate of the reward function with the corresponding value function, modulated by a $ extit{sign}$ to indicate whether the optimism or pessimism is chosen. VPO also directly optimizes the policy with implicit reward modeling, and therefore shares a simpler RLHF pipeline similar to direct preference optimization. Theoretical guarantees of VPO are provided for both online and offline settings, matching the rates of their standard RL counterparts. Moreover, experiments on text summarization and dialog verify the practicality and effectiveness of VPO.

Problem

• Unified online and offline RLHF
• Incorporating uncertainty in reward
• Optimizing policy with implicit modeling

Innovation

• Unified RLHF approach
• Value-incentivized preference optimization
• Implicit reward modeling