Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity

2024-06-10

arXiv.org

AI Summary

In preference-based reinforcement learning (PbRL), human feedback often contains abundant irrelevant information, rendering reward models vulnerable to noise. To address this, we propose R2N, the first PbRL framework incorporating dynamic sparsity training. R2N adaptively prunes neural network weights during training to suppress responses to irrelevant state features while enhancing selective modeling of task-relevant features. It unifies preference modeling, inverse reward inference, and a simulated teacher feedback mechanism to achieve robust reward function learning. Evaluated across diverse simulated robotic tasks, R2N consistently outperforms existing sparse training and PbRL methods, demonstrating superior stability and reward modeling accuracy—particularly under high-noise conditions. This work establishes a novel paradigm for robust, human-in-the-loop autonomous decision-making.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Machine Learning: Reinforcement Learning
• Humans and AI: Learning Human Values and Preferences

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

To integrate into human-centered environments, autonomous agents must learn from and adapt to humans in their native settings. Preference-based reinforcement learning (PbRL) can enable this by learning reward functions from human preferences. However, humans live in a world full of diverse information, most of which is irrelevant to completing any particular task. It then becomes essential that agents learn to focus on the subset of task-relevant state features. To that end, this work proposes R2N (Robust-to-Noise), the first PbRL algorithm that leverages principles of dynamic sparse training to learn robust reward models that can focus on task-relevant features. In experiments with a simulated teacher, we demonstrate that R2N can adapt the sparse connectivity of its neural networks to focus on task-relevant features, enabling R2N to significantly outperform several sparse training and PbRL algorithms across simulated robotic environments.

Problem

• Enhancing robustness in preference-based reinforcement learning
• Focusing on task-relevant state features dynamically
• Improving reward models with dynamic sparse training

Innovation

• Dynamic sparse training for robust reward models
• Focus on task-relevant state features
• Adapt neural network sparse connectivity dynamically