ActiveUMI: Robotic Manipulation with Active Perception from Robot-Free Human Demonstrations

2025-10-01

AI Summary

To address low-quality demonstration data, poor generalization, and the absence of active egocentric perception modeling in transferring real-world human demonstrations to bimanual robots, this paper proposes ActiveUMI. Our method introduces a novel VR teleoperation system integrated with head motion tracking to achieve kinematic alignment between human and robot while jointly modeling active visual attention and manipulation behavior. We further design a lightweight, end-to-end mobile data collection platform incorporating immersive 3D rendering, wearable edge computing, and efficient online calibration. Evaluated on six bimanual manipulation tasks, policies trained exclusively on ActiveUMI-collected data achieve an average success rate of 70%. Crucially, they retain strong generalization—achieving 56% success when deployed on unseen objects and novel environments—demonstrating substantial improvements in cross-domain adaptability and robustness.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We present ActiveUMI, a framework for a data collection system that transfers in-the-wild human demonstrations to robots capable of complex bimanual manipulation. ActiveUMI couples a portable VR teleoperation kit with sensorized controllers that mirror the robot's end-effectors, bridging human-robot kinematics via precise pose alignment. To ensure mobility and data quality, we introduce several key techniques, including immersive 3D model rendering, a self-contained wearable computer, and efficient calibration methods. ActiveUMI's defining feature is its capture of active, egocentric perception. By recording an operator's deliberate head movements via a head-mounted display, our system learns the crucial link between visual attention and manipulation. We evaluate ActiveUMI on six challenging bimanual tasks. Policies trained exclusively on ActiveUMI data achieve an average success rate of 70% on in-distribution tasks and demonstrate strong generalization, retaining a 56% success rate when tested on novel objects and in new environments. Our results demonstrate that portable data collection systems, when coupled with learned active perception, provide an effective and scalable pathway toward creating generalizable and highly capable real-world robot policies.

Problem

• Transferring human demonstrations to robots for complex bimanual manipulation tasks
• Capturing active egocentric perception linking visual attention to manipulation
• Creating generalizable robot policies through portable data collection systems

Innovation

• Portable VR teleoperation with sensorized robot-mirroring controllers
• Active egocentric perception via head movement recording
• Self-contained wearable system with immersive 3D rendering