UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning

2026-06-17

AI Summary

This work addresses the poor sample efficiency of preference-based reinforcement learning under passive data collection, particularly in early learning stages. The authors propose a model-based active exploration method that unifies the uncertainty modeling of rewards, environment dynamics, and value functions within a preference-aware planning objective, thereby eliminating the need for heuristic exploration designs. By leveraging ensemble-based uncertainty estimates and incorporating them into a weighted trajectory scoring mechanism, the approach provides sublinear regret guarantees in both finite and infinite horizons. Empirical results on the Meta-World benchmark demonstrate substantial improvements over existing model-free preference methods and non-optimistic model-based baselines, achieving significantly higher sample efficiency.

Abstract

Preference-based RL provides an approach to learning reward models from pairwise comparisons of behaviors, bypassing the need for explicit reward design. However, existing methods typically rely on passive data collection and suffer from poor sample efficiency, especially during the early stages of learning. We introduce a model-based approach that actively directs exploration by jointly reasoning over uncertainties in the reward, dynamics, and value functions. Our method, Uncertainty-Balanced Preference Planning (UBP2), uses ensembles of reward, dynamics, and value function models to evaluate candidate trajectories according to a unified score that combines expected reward, terminal value, and epistemic uncertainty. Planning under this objective yields an explicit tradeoff between exploitation and information acquisition without requiring ad hoc exploration heuristics. Under standard regularity assumptions, we establish sublinear regret guarantees for both finite-horizon and infinite-horizon settings. Empirically, experiments on the Meta-World benchmark show UBP2 achieves substantially higher sample efficiency than model-free preference-based methods and non-optimistic model-based baselines.

Problem

• preference-based reinforcement learning
• sample efficiency
• active exploration
• reward modeling
• uncertainty quantification

Innovation

• preference-based reinforcement learning
• uncertainty quantification
• active exploration
• model-based planning
• sample efficiency