JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling

2026-08-10

AI Summary

This work addresses the high deployment cost of existing video-based generative world models and the suboptimal control performance caused by decoupling state prediction from action modeling in latent approaches. It introduces, for the first time, the Joint-Embedding Predictive Architecture (V-JEPA) to world modeling, constructing a shared predictor in the pretrained V-JEPA latent space that jointly learns state transitions and continuous action generation. A structured current–future joint objective preserves dense visual correspondences, enabling tight coupling between prediction and policy. The resulting model integrates seamlessly into vision–language–action (VLA) systems. Evaluated on LIBERO-Plus, it achieves a 79.2% success rate—the best among methods without large-scale robot pretraining—and reaches a new state-of-the-art overall performance of 86.3% with the π₀.₅ instantiation, while demonstrating strong generalization on RoboTwin 2.0 and real-world dual-arm tasks.

Abstract

Robust robot control benefits from explicitly modeling state transitions, but video-generation world action models (WAMs) introduce substantial deployment cost. Existing latent WAMs avoid explicit future generation, but often compress predictive representations or separate predictive modeling from the representations used for action generation. We introduce JEPA-WAM, a latent WAM built in a pretrained V-JEPA space, which couples latent transition prediction with continuous action generation through a shared predictor. JEPA-WAM predicts a spatially structured joint current-future target that captures task-shared visual temporal structure between current and future observations, while preserving dense patch-level correspondence. Through the shared predictor, transition supervision directly shapes the backbone, from which dedicated representations are extracted for action prediction. The same design can also be instantiated in pretrained VLA policies while preserving their original perception and action pathways. On LIBERO-Plus, JEPA-WAM achieves 79.2%, the best result without large-scale robot-policy pretraining, while its pretrained $π_{0.5}$ instantiation reaches 86.3%, achieving the best overall performance. Experiments on RoboTwin 2.0 and real-world bimanual manipulation further demonstrate strong generalization under visual and spatial shifts.

Problem

• world action models
• latent representation
• vision-language-action policies
• robot control
• predictive modeling

Innovation

• JEPA-WAM
• latent world model
• joint-embedding prediction
• vision-language-action policy
• spatially structured forecasting