Latent Action Pretraining from Videos

2024-10-15

International Conference on Learning Representations

AI Summary

Existing vision-language-action (VLA) models rely heavily on human teleoperation annotations for action labels, severely limiting scalability and generalization. This paper introduces the first unsupervised pretraining framework for VLA models using internet-scale videos *without* action labels: it leverages VQ-VAE to quantize action representations in latent space, generating pseudo-action labels; employs a multimodal Transformer to predict these latent actions; and adopts a two-stage transfer paradigm—video pretraining followed by robot fine-tuning. To our knowledge, this is the first method enabling VLA pretraining solely from human demonstration videos devoid of ground-truth action signals. Evaluated on real-world robotic tasks—including language-conditioned manipulation, zero-shot object generalization, and zero-shot instruction semantics—the approach outperforms all prior video-driven methods and significantly surpasses state-of-the-art VLA models trained with real action labels.

Technology Category

• Computer Vision: Large Vision Models
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We introduce Latent Action Pretraining for general Action models (LAPA), an unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels. Existing Vision-Language-Action models require action labels typically collected by human teleoperators during pretraining, which significantly limits possible data sources and scale. In this work, we propose a method to learn from internet-scale videos that do not have robot action labels. We first train an action quantization model leveraging VQ-VAE-based objective to learn discrete latent actions between image frames, then pretrain a latent VLA model to predict these latent actions from observations and task descriptions, and finally finetune the VLA on small-scale robot manipulation data to map from latent to robot actions. Experimental results demonstrate that our method significantly outperforms existing techniques that train robot manipulation policies from large-scale videos. Furthermore, it outperforms the state-of-the-art VLA model trained with robotic action labels on real-world manipulation tasks that require language conditioning, generalization to unseen objects, and semantic generalization to unseen instructions. Training only on human manipulation videos also shows positive transfer, opening up the potential for leveraging web-scale data for robotics foundation model.

Problem

• Unsupervised pretraining for Vision-Language-Action models without action labels
• Learning discrete latent actions from internet-scale unlabeled videos
• Improving robot manipulation via language-conditioned semantic generalization

Innovation

• Unsupervised pretraining without action labels
• VQ-VAE-based discrete latent action learning
• Latent VLA model for action prediction