GAINS: Leveraging Inconsistent Human Intervention Signals in Reinforcement Learning

2026-08-16

AI Summary

This study addresses the challenge of policy learning in reinforcement learning caused by delayed and inconsistent human intervention signals. We propose GAINS, a framework that systematically models the temporal inconsistency of human interventions for the first time. By integrating distributional reinforcement learning with pessimistic exploration strategies, GAINS effectively characterizes reward variability induced by imperfect interventions, enabling safe and efficient policy correction. Experiments demonstrate that GAINS outperforms RLIF in both simulated and real-world robotic tasks, achieving a 22% increase in success rate and a 43% improvement in fault recovery success rate. These results indicate that GAINS significantly enhances sample efficiency and safety in human-in-the-loop learning scenarios by robustly handling the stochasticity inherent in delayed human feedback.

Abstract

Correcting robot manipulation policies through human intervention holds great promise for real-world deployment, yet human operators are inherently imperfect in both the actions they provide and the timing of their intervention signals. While the former has been extensively discussed in reinforcement learning (RL), the latter remains underexplored. At high control frequencies, human intervention signals are often delayed and inconsistent across time and state space. In this work, we present GAINS, a framework for leveraging inconsistent human intervention signals in RL. At the core of GAINS, we employ distributional RL with quantile Q-networks to model the return variability induced by sparse task rewards and inconsistent human interventions. Building on this distributional representation, we introduce a pessimistic exploration strategy that promotes safe and sample-efficient learning under human corrections. We evaluate GAINS on four diverse simulated manipulation tasks and two challenging real-world scenarios against state-of-the-art intervention-based methods. GAINS achieves a 22% higher task success rate than RLIF and improves recovery success by up to 43% in failure scenarios. These results highlight the importance of modeling return variability induced by human imperfection for real-world deployment of intervention-based learning.

Problem

• Reinforcement Learning
• Human Intervention
• Inconsistent Signals
• Robot Manipulation

Innovation

• Inconsistent Human Intervention
• Distributional Reinforcement Learning
• Quantile Q-networks
• Pessimistic Exploration
• Return Variability Modeling