ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining

2026-06-15

AI Summary

This work addresses the significant discrepancies between heterogeneous egocentric human videos and robot trajectories in terms of action spaces, body morphology, temporal dynamics, and supervision quality by proposing a unified vision–language–action (VLA) pretraining framework. It introduces a scalable video-to-action pipeline to generate robot-compatible pseudo-action trajectories and employs camera-space action representations, morphology-conditioned modeling, and temporally aligned chunking to achieve cross-domain alignment. A reliability-aware loss further enhances training robustness. This approach enables, for the first time, efficient joint pretraining on large-scale egocentric human videos and robotic data, achieving state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, and demonstrating exceptional cross-domain transfer capabilities in real-world bimanual manipulation tasks.

Abstract

Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive. Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining. However, joint training on human and robot data remains challenging due to divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality. We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources. To extract large-scale pretraining supervision from egocentric human videos, we build a scalable egocentric video-to-action pipeline that converts raw human videos into robot-format pseudo-action trajectories. To make these labels comparable with robot demonstrations, ACE-EGO-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. To robustly leverage noisy pseudo-action supervision from egocentric human videos, we formulate a reliability-aware training objective with a human auxiliary loss that concentrates supervision on reliable signals. We instantiate ACE-EGO-0 on 4.53K hours of robot and simulation data, together with 1.48K hours of pseudo-action-labeled egocentric human data. Experiments show that incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-EGO-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, while demonstrating strong transfer to real-world bimanual manipulation.

Problem

• Vision-Language-Action
• egocentric human videos
• robot trajectory
• unified pretraining
• action representation

Innovation

• Vision-Language-Action
• egocentric video
• unified action representation
• pseudo-action trajectory
• reliability-aware training