General Exploratory Bonus for Optimistic Exploration in RLHF

2025-09-27

AI Summary

In RLHF, existing KL- or α-divergence-based exploration incentives suffer from regularization bias, steering policies toward high-probability regions of the reference model and resulting in overly conservative behavior and insufficient exploration. To address this, we propose Generalized Exploration Bonus (GEB), the first theoretically grounded method ensuring optimistic exploration: GEB explicitly counteracts divergence-induced bias via reference-dependent reward shaping, unifying and generalizing multiple heuristic exploration rewards. Crucially, GEB is compatible with the full α-divergence family of regularizers, enabling flexible exploration mechanism design. Extensive experiments across diverse alignment tasks—including instruction following, math reasoning, and safety alignment—and across language models of varying scales demonstrate that GEB significantly improves both sample efficiency and final performance. These results empirically validate the theoretical guarantees of GEB, confirming its effectiveness and practical utility in real-world RLHF settings.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Optimistic exploration is central to improving sample efficiency in reinforcement learning with human feedback, yet existing exploratory bonus methods to incentivize exploration often fail to realize optimism. We provide a theoretical analysis showing that current formulations, under KL or $alpha$-divergence regularization, unintentionally bias exploration toward high-probability regions of the reference model, thereby reinforcing conservative behavior instead of promoting discovery of uncertain regions. To address this pitfall, we introduce the General Exploratory Bonus (GEB), a novel theoretical framework that provably satisfies the optimism principle. GEB counteracts divergence-induced bias via reference-dependent reward regulation and unifies prior heuristic bonuses as special cases, while extending naturally across the full $alpha$-divergence family. Empirically, GEB consistently outperforms baselines on alignment tasks across multiple divergence settings and large language model backbones. These results demonstrate that GEB offers both a principled and practical solution for optimistic exploration in RLHF.

Problem

• Addresses biased exploration in RLHF due to divergence regularization
• Introduces General Exploratory Bonus framework for optimistic exploration
• Improves alignment performance across divergence settings and model architectures

Innovation

• GEB framework counteracts divergence-induced bias in RLHF
• GEB unifies prior heuristic bonuses as special cases
• GEB extends naturally across full α-divergence family