OHP-RL: Online Human Preference as Guidance in Reinforcement Learning for Robot Manipulation

2026-05-15

AI Summary

This work addresses the challenges of deploying reinforcement learning on real-world robotic systems, where inefficient and unsafe exploration hinders practical application, and existing approaches fail to effectively leverage preference information embedded in human interventions. To overcome these limitations, the authors propose a state-dependent adaptive preference gating mechanism that, for the first time, models human intervention as state-specific relative preferences rather than action demonstrations, dynamically modulating the influence of human feedback on policy learning. Integrating online preference learning, reinforcement learning optimization, and a gating architecture, the method is evaluated on a Franka robot across multiple contact-rich manipulation tasks. Experimental results demonstrate that, compared to baseline methods, the proposed approach significantly improves learning efficiency and safety, achieving higher task success rates, faster convergence, reduced human intervention, and more stable, human-aligned policy behavior.

Abstract

While reinforcement learning (RL) enables robots to acquire skills autonomously, its real-world deployment is severely limited by inefficient and unsafe exploration. Human-in-the-loop interventions offer a practical solution, yet existing methods typically exploit these interventions as auxiliary training signals, without fully capturing the richer information they provide about when and how autonomy should be guided. Human interventions often encode relative preferences over behavior under safety and task constraints, rather than prescribing exact actions to imitate. Motivated by this perspective, we propose Online Human Preference as Guidance in Reinforcement Learning (OHP-RL), a framework that leverages human interventions as preference information to guide policy learning. OHP-RL introduces a state-dependent preference gate that adaptively regulates when and to what extent human interventions should shape policy learning. This design enables the agent to benefit from intermittent and imperfect human feedback while preserving autonomous exploration and stable policy optimization. We evaluate OHP-RL on three challenging real-world contact-rich manipulation tasks on a Franka robot. Across all tasks, OHP-RL consistently achieves strong success rates, faster convergence, and substantially lower human intervention effort than prior approaches. Moreover, the learned policies exhibit more stable and human-aligned behavior throughout training.

Problem

• reinforcement learning
• human preference
• robot manipulation
• safe exploration
• human-in-the-loop

Innovation

• human-in-the-loop reinforcement learning
• preference-based guidance
• state-dependent gating
• robot manipulation
• online human feedback