Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis

2024-06-11

RLJ

AI Summary

This work addresses reward-free reinforcement learning from human feedback (RLHF), focusing on direct optimal policy learning under trajectory-level preferences. We propose BSAD—the first reward-model-free, model-agnostic RLHF algorithm—integrating backward dynamic programming, a dueling bandit subroutine, and reward-agnostic exploration, augmented with an adaptive stopping criterion and a frame-based discounted MDP extension. Theoretically, we establish the first instance-dependent sample complexity upper bound of Õ(cₘSA³H³M log(1/δ)), proving RLHF is no harder than classical RL; we further show end-to-end learning avoids overfitting and distributional shift inherent in two-stage approaches that rely on reward modeling. Empirically, BSAD achieves logarithmic regret and significantly outperforms conventional two-stage RLHF methods.

Technology Category

• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Online Learning & Bandits

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In this paper, we study reinforcement learning from human feedback (RLHF) under an episodic Markov decision process with a general trajectory-wise reward model. We developed a model-free RLHF best policy identification algorithm, called $mathsf{BSAD}$, without explicit reward model inference, which is a critical intermediate step in the contemporary RLHF paradigms for training large language models (LLM). The algorithm identifies the optimal policy directly from human preference information in a backward manner, employing a dueling bandit sub-routine that constantly duels actions to identify the superior one. $mathsf{BSAD}$ adopts a reward-free exploration and best-arm-identification-like adaptive stopping criteria to equalize the visitation among all states in the same decision step while moving to the previous step as soon as the optimal action is identifiable, leading to a provable, instance-dependent sample complexity $ ilde{mathcal{O}}(c_{mathcal{M}}SA^3H^3Mlogfrac{1}{delta})$ which resembles the result in classic RL, where $c_{mathcal{M}}$ is the instance-dependent constant and $M$ is the batch size. Moreover, $mathsf{BSAD}$ can be transformed into an explore-then-commit algorithm with logarithmic regret and generalized to discounted MDPs using a frame-based approach. Our results show: (i) sample-complexity-wise, RLHF is not significantly harder than classic RL and (ii) end-to-end RLHF may deliver improved performance by avoiding pitfalls in reward inferring such as overfit and distribution shift.

Problem

• Reinforcement Learning
• Human Feedback
• Reward Function

Innovation

• BSAD algorithm
• RLHF method
• human feedback learning