HALOMI: Learning Humanoid Loco-Manipulation with Active Perception from Human Demonstrations

2026-06-17

AI Summary

This work addresses the challenges of out-of-distribution tracking fragility and the human–robot gap that arise when humanoid robots learn from human demonstrations, primarily due to discrepancies in perception and actuation. To bridge this gap, the authors propose a scalable learning framework that fuses first-person and wrist-mounted visual perspectives. The approach integrates a manifold-constrained controller, latent behavioral manifold planning, cross-view alignment, and a controller-aware reference trajectory adaptation mechanism, enabling robust head-to-hand world-coordinate tracking. Leveraging an extended UMI system for multi-view data collection, the method is validated on a Unitree G1 humanoid robot across five real-world tasks. It achieves an average success rate of 85% on three quantitative benchmarks and successfully demonstrates complex behaviors such as dynamic throwing and squat-and-grasp maneuvers.

Abstract

Human demonstrations, which can be collected at scale and naturally capture active hand-eye coordination, are a promising data source for learning humanoid loco-manipulation. However, directly transferring human demonstrations to humanoids requires a precise world-frame tracking controller, which is often brittle under Out-of-Distribution(OOD) targets, while human-to-humanoid gaps persist in both egocentric observation and action execution. To address these challenges, we present HALOMI, a scalable framework for learning humanoid loco-manipulation with active perception from human demonstrations. HALOMI extends Universal Manipulation Interface (UMI) with egocentric sensing to collect ego-view and wrist-view observations along with head-hand trajectories at scale. We further propose a manifold-constrained controller that plans in a learned latent behavior manifold to enable precise and robust head-hand tracking in the world frame. To bridge the human-to-humanoid gap, we perform ego-view alignment and introduce a controller-aware reference trajectory adaptation to reduce mismatch in both observation and action execution. We validate HALOMI on a Unitree G1 humanoid robot with an actuated neck across five real-world tasks involving navigation, grasping, bimanual manipulation, whole-body coordination, and dynamic behaviors. Across the three quantitatively evaluated tasks, HALOMI achieves an average success rate of 85\%, while additional qualitative demonstrations show its ability to support dynamic tossing and deep-squat grasping.

Problem

• humanoid loco-manipulation
• human demonstrations
• active perception
• human-to-humanoid gap
• out-of-distribution targets

Innovation

• humanoid loco-manipulation
• active perception
• egocentric sensing
• manifold-constrained control
• demonstration transfer