ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learning

2026-06-15

AI Summary

This work addresses the challenge in humanoid robot manipulation where human interventions—often necessitated by system complexity—introduce low-quality trajectories that degrade traditional imitation learning by propagating inefficient or erroneous behaviors. To overcome this, the authors propose the ROVE framework, which innovatively integrates human-in-the-loop data collection, Optimistic Value Estimation (OVE), cross-embodiment supervision from human videos, and advantage-guided policy updates to distill high-value actions from mixed-quality interventions. By effectively filtering out suboptimal demonstrations and emphasizing rare failure-and-recovery patterns, ROVE significantly outperforms existing baselines on real-world fine manipulation tasks and demonstrates continual performance gains across multiple rounds of human–robot interaction.

Abstract

Human interventions provide crucial corrective signals for post-training Vision-Language-Action (VLA) models. However, enabling seamless humanoid interventions is a formidable systems challenge due to complex whole-body kinematics and dexterous-hand control. Consequently, the collected intervention trajectories are often suboptimal, and methods that rely on human interventions as expert supervision can absorb hesitant, inefficient, or even erroneous behaviors. To address both the system and algorithmic challenges, we propose ROVE, a reinforcement learning framework for humanoid VLA post-training with imperfect human interventions. First, ROVE introduces a human-in-the-loop pipeline capable of collecting deployment and intervention data for humanoid manipulation. Second, it utilizes Optimistic Value Estimation (OVE) to prioritize high-value behaviors from mixed-quality trajectories. To further robustify value estimation, we incorporate cross-embodiment human experience videos to provide rich supervision for long-tailed failure and recovery modes. The resulting critic yields informative advantage signals, steering the VLA actor to focus on high-value behaviors rather than indiscriminately imitating all actions. On challenging real-world contact-rich and fine-grained humanoid manipulation tasks, ROVE outperforms experience-learning baselines and consistently improves across multiple rollout-intervention iterations.

Problem

• humanoid manipulation
• human interventions
• Vision-Language-Action models
• imperfect supervision
• reinforcement learning

Innovation

• Reinforcement Learning
• Human-in-the-loop
• Optimistic Value Estimation
• Vision-Language-Action Models
• Humanoid Manipulation