Motion-Focused Latent Action Enables Cross-Embodiment VLA Training from Human EgoVideos

2026-06-17

AI Summary

This work addresses the limitation of current vision–language–action (VLA) models, which struggle to leverage the vast amounts of unlabeled human egocentric demonstration videos due to the scarcity of large-scale robot action-labeled datasets. The authors propose a latent-action pretraining framework that extracts generalizable action priors from unlabeled human videos by combining a hybrid disentangled VQ-VAE with physics-informed masking to separate motion from background. This enables the construction of a cross-embodiment action codebook. Coupled with an intention-aware disentanglement strategy, the approach adapts to specific robotic platforms using as few as 50 task trajectories. Notably, this is the first method to pretrain VLA models exclusively on unlabeled human videos while achieving performance on par with state-of-the-art approaches that rely on extensive annotated robot data, in both simulation and real-world environments.

Abstract

Training generalist Vision-Language-Action(VLA) models typically requires massive, diverse robotic datasets with high-fidelity action annotations. While egocentric human manipulation videos are abundant and capture significant environmental diversity, the absence of action labels makes them difficult to use in conventional training paradigms. To address this, we propose a latent-action-based framework designed to extract general action priors from unlabeled human videos. The architecture features a Hybrid Disentangled VQ-VAE that decouples motion dynamics from environmental backgrounds through physical masks, enabling the construction of a cross-embodiment action codebook. By pre-training on human videos with the codebook, the VLM backbone learns deep representations of action intent. For adaptation to specific embodiments, we introduce an intent-perception decoupling strategy where the VLM predicts the action intent while a separate frozen visual encoder provides state-specific features to the action expert, thereby reducing action hallucinations. Results in simulation and real-world environments show that our method, pre-trained exclusively on unlabeled human videos, performs competitively with state-of-the-art VLA models trained on massive annotated datasets, requiring only 50 trajectories for downstream adaptation.

Problem

• Vision-Language-Action
• egocentric videos
• action labels
• cross-embodiment
• robotic learning

Innovation

• latent action
• cross-embodiment
• VQ-VAE
• egocentric video
• intent-perception decoupling