Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

2026-07-02

AI Summary

Vision-Language-Action models are hindered by the scarcity of expert demonstration data, making large-scale collection of observation-instruction-action triplets challenging. This work proposes a Task-Agnostic Pretraining (TAP) framework that decouples physical skill acquisition from semantic alignment for the first time: it first learns a motion prior through inverse dynamics self-supervision on unlabeled interaction data, followed by lightweight language grounding fine-tuning using only minimal expert demonstrations. This approach drastically reduces reliance on expert data, achieving performance on the SIMPLER benchmark comparable to models trained on millions of expert trajectories while using only a tiny fraction of labeled examples—yielding an absolute success rate improvement of 10%. On the real-world WidowX platform, it maintains a 25% success rate under camera perturbations, substantially outperforming existing baselines.

Abstract

Vision-Language-Action (VLA) models are fundamentally bottlenecked by the scarcity of expert demonstrations -- triplets of observations, instructions, and actions that are costly to collect at scale. We argue that this bottleneck stems from conflating two distinct learning objectives: acquiring physical competence (how to move) and acquiring semantic alignment (what to do). Crucially, only the latter requires language supervision. Building on this Decomposition Hypothesis, we propose Task-Agnostic Pretraining (TAP), a two-stage framework that first learns transferable motor priors from cheap, unlabeled interaction data -- including discarded off-task trajectories and autonomous robot play -- via a self-supervised Inverse Dynamics objective. A lightweight second stage then grounds these priors in language using minimal expert data. On the SIMPLER benchmark, TAP matches models trained on over 1M expert trajectories while using orders of magnitude less labeled data, yielding a 10% absolute gain over standard behavior cloning. On a real-world WidowX platform, TAP retains 25% success under camera perturbations where internet-scale baselines collapse to 0%, demonstrating that task-agnostic pretraining produces robust, transferable physical representations and offers a scalable path forward for Embodied AI.

Problem

• Vision-Language-Action
• expert demonstrations
• data scarcity
• embodied AI
• physical competence

Innovation

• Task-Agnostic Pretraining
• Vision-Language-Action
• Inverse Dynamics
• Self-Supervised Learning
• Embodied AI