S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning

2024-08-22

AI Summary

In preference-based reinforcement learning (PbRL), inefficient policy learning arises when trajectory segments lack distinguishability due to ambiguous contextual cues. To address this, we propose Skill-Enhanced Preference Optimization Algorithm (S-EPOA), the first PbRL framework integrating a skill-driven active querying mechanism. S-EPOA jointly optimizes information gain and trajectory discriminability within a pre-trained skill embedding space, overcoming the identifiability bottleneck inherent in conventional PbRL. Technically, it unifies unsupervised skill pretraining, skill-space modeling, and uncertainty-aware query selection. Evaluated on robotic manipulation and locomotion control tasks, S-EPOA achieves an average 42% faster convergence and reduces preference annotation requirements by 35% compared to state-of-the-art PbRL methods, significantly improving both learning efficiency and policy robustness.

Technology Category

• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Preference-based reinforcement learning (PbRL) stands out by utilizing human preferences as a direct reward signal, eliminating the need for intricate reward engineering. However, despite its potential, traditional PbRL methods are often constrained by the indistinguishability of segments, which impedes the learning process. In this paper, we introduce Skill-Enhanced Preference Optimization Algorithm (S-EPOA), which addresses the segment indistinguishability issue by integrating skill mechanisms into the preference learning framework. Specifically, we first conduct the unsupervised pretraining to learn useful skills. Then, we propose a novel query selection mechanism to balance the information gain and distinguishability over the learned skill space. Experimental results on a range of tasks, including robotic manipulation and locomotion, demonstrate that S-EPOA significantly outperforms conventional PbRL methods in terms of both robustness and learning efficiency. The results highlight the effectiveness of skill-driven learning in overcoming the challenges posed by segment indistinguishability.

Problem

• Preference-based Reinforcement Learning
• Context Differentiation
• Learning Efficiency

Innovation

• Skill-Enhanced Preference Optimization Algorithm (S-EPOA)
• Skill Learning
• Problem Selection Strategy