Preference-Guided Reinforcement Learning for Efficient Exploration

2024-07-09

arXiv.org

AI Summary

For reinforcement learning tasks characterized by sparse rewards, long horizons, and poor explorability, existing preference-based RL (PbRL) methods suffer from inefficient exploration due to their reliance on explicit reward modeling. This paper proposes LOPE, the first framework that directly leverages human trajectory preferences for *online* policy exploration—bypassing reward modeling entirely. Its core innovations are: (1) a preference-guided mechanism enforcing trajectory-level state marginal matching, and (2) a two-step sequential policy optimization—comprising trust-region–constrained improvement followed by preference alignment—with provable performance improvement bounds. Experiments across diverse hard-exploration environments demonstrate that LOPE significantly accelerates convergence and improves final policy performance, consistently outperforming state-of-the-art preference-based and sparse-reward RL methods.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

In this paper, we investigate preference-based reinforcement learning (PbRL), which enables reinforcement learning (RL) agents to learn from human feedback. This is particularly valuable when defining a fine-grain reward function is not feasible. However, this approach is inefficient and impractical for promoting deep exploration in hard-exploration tasks with long horizons and sparse rewards. To tackle this issue, we introduce LOPE: extbf{L}earning extbf{O}nline with trajectory extbf{P}reference guidanc extbf{E}, an end-to-end preference-guided RL framework that enhances exploration efficiency in hard-exploration tasks. Our intuition is that LOPE directly adjusts the focus of online exploration by considering human feedback as guidance, thereby avoiding the need to learn a separate reward model from preferences. Specifically, LOPE includes a two-step sequential policy optimization technique consisting of trust-region-based policy improvement and preference guidance steps. We reformulate preference guidance as a trajectory-wise state marginal matching problem that minimizes the maximum mean discrepancy distance between the preferred trajectories and the learned policy. Furthermore, we provide a theoretical analysis to characterize the performance improvement bound and evaluate the effectiveness of the LOPE. When assessed in various challenging hard-exploration environments, LOPE outperforms several state-of-the-art methods in terms of convergence rate and overall performance.The code used in this study is available at https://github.com/buaawgj/LOPE.

Problem

• Addresses inefficient exploration in preference-based reinforcement learning tasks
• Solves hard-exploration problems with long horizons and sparse rewards
• Eliminates need for separate reward model learning from human preferences

Innovation

• Online preference-guided RL framework for exploration
• Two-step sequential policy optimization with trust regions
• Trajectory-wise state marginal matching for preference guidance