DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition

2025-05-09

AI Summary

This work addresses the low query efficiency of preference-based reinforcement learning (PbRL) in robotic skill acquisition, identifying its root cause as policy bias—leading to insufficient trajectory diversity and diminished human preference discriminability. To resolve this, we first introduce a *preference discriminability metric*, then propose a *multi-policy parallel training framework*, and design a *discriminator-driven discriminative query sampling mechanism*. These components jointly optimize reward modeling and discriminability objectives, augmented by trajectory diversity regularization. Evaluated on both simulated and real-world quadrupedal robot tasks, our method significantly improves query efficiency: it achieves over 30% improvement over state-of-the-art approaches under low-discriminability preference settings. The approach establishes a new paradigm for efficient and robust human-feedback-driven learning in robotics.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Preference-based Reinforcement Learning (PbRL) enables policy learning through simple queries comparing trajectories from a single policy. While human responses to these queries make it possible to learn policies aligned with human preferences, PbRL suffers from low query efficiency, as policy bias limits trajectory diversity and reduces the number of discriminable queries available for learning preferences. This paper identifies preference discriminability, which quantifies how easily a human can judge which trajectory is closer to their ideal behavior, as a key metric for improving query efficiency. To address this, we move beyond comparisons within a single policy and instead generate queries by comparing trajectories from multiple policies, as training them from scratch promotes diversity without policy bias. We propose Discriminability-Aware Policy-to-Policy Preference-Based Efficient Reinforcement Learning (DAPPER), which integrates preference discriminability with trajectory diversification achieved by multiple policies. DAPPER trains new policies from scratch after each reward update and employs a discriminator that learns to estimate preference discriminability, enabling the prioritized sampling of more discriminable queries. During training, it jointly maximizes the preference reward and preference discriminability score, encouraging the discovery of highly rewarding and easily distinguishable policies. Experiments in simulated and real-world legged robot environments demonstrate that DAPPER outperforms previous methods in query efficiency, particularly under challenging preference discriminability conditions.

Problem

• Improves query efficiency in preference-based reinforcement learning
• Enhances trajectory diversity by comparing multiple policies
• Maximizes preference reward and discriminability for robot skills

Innovation

• Compares trajectories from multiple policies
• Trains new policies from scratch
• Prioritizes sampling discriminable queries