EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal

2026-07-15

AI Summary

This work addresses the limited environmental context awareness that hinders humanoid robots from robust locomotion on unstructured terrains. To bridge this gap, the authors present a multi-sensor system integrating first-person wearable devices and a portable 3D scanner to capture and generate scene-aligned 4D human motion sequences. They introduce EgoHTR, a large-scale dataset comprising over 150,000 frames focused on complex terrains, which fills a critical void between human motion learning and scene reconstruction. Leveraging this dataset, they develop a gait controller endowed with environmental perception through multi-sensor fusion, 4D motion capture, and reinforcement learning. The controller is successfully deployed on the Unitree G1 robot, enabling stable traversal across real-world challenging terrains and achieving state-of-the-art motion accuracy.

Abstract

Deploying humanoid robots in unstructured terrain remains an open problem. While classic reinforcement learning struggles with the sheer complexity of real-world interactions, more promising methods leveraging human priors remain limited to models lacking contextual awareness. The restricted motion synthesis is a direct consequence of existing dataset pipelines failing to capture human-scene sequences in challenging environments. To bridge this gap between humanoid learning and scene reconstruction, we introduce the Egocentric Human-Terrain Reconstruction (EgoHTR) dataset. We develop and open-source a reconstruction pipeline capturing 55 scene-aligned 4D human motion sequences in diverse, complex environments using a multi-sensor setup of egocentric wearables and a portable 3D scanner. The resulting dataset comprises over 150k frames, which we evaluate against motion-capture ground truth, demonstrating state-of-the-art accuracy and establishing a rigorous benchmark for human motion analysis and synthesis. Further, we leverage this data to train perceptive locomotion policies, demonstrating hardware deployment on a Unitree G1 for reconstructed reference motions. Our pipeline enables community-driven dataset extensions and factors the problem to help researchers build foundational, context-aware robots that reliably traverse uneven terrain.

Problem

• humanoid robots
• unstructured terrain
• egocentric 4D demonstrations
• human-scene interaction
• motion synthesis

Innovation

• Egocentric 4D reconstruction
• humanoid locomotion
• terrain-aware motion synthesis
• multi-sensor data pipeline
• perceptive robot policy