Vid2WAM: Distilling Video Diffusion Priors into World Action Models

2026-08-09

AI Summary

This work addresses the limited scalability and generalization of World Action Models (WAMs), which rely heavily on costly expert demonstrations. The authors propose Vid2WAM, a novel framework that, for the first time, transfers the rich generic priors from large-scale video diffusion models to compact WAMs via offline distillation. By jointly leveraging language-conditioned future prediction and an inverse dynamics model to generate pseudo-actions, Vid2WAM enables effective supervision without requiring expert trajectories for target tasks. A key innovation is the introduction of a source-aware residual action adaptation mechanism that seamlessly integrates synthetic and real supervision signals. Experiments demonstrate that Vid2WAM significantly improves generalization to novel tasks and data efficiency in both simulation and real-world environments, enabling low-latency, efficient deployment—even with scarce expert demonstrations.

Abstract

World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions. However, their scalability and generalization remain constrained by their reliance on costly expert demonstrations. We challenge this by asking whether future supervision for WAMs must originate from target-task expert trajectories. In this paper, we propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. Given an observation and language instruction, Vid2WAM distills supervision through two complementary channels: task-conditioned future rollouts directly supervise the student's future prediction branch, while an inverse dynamics model recovers embodiment-specific pseudo-actions for action learning. To robustly integrate synthetic and real supervision, we introduce source-aware residual action adaptation that learns source-specific corrections around a shared action backbone and mitigates interference from noisy pseudo-actions. During inference, both the video teacher and inverse dynamics model are discarded, leaving only the WAM student for efficient deployment. Simulation and real-world experiments demonstrate that Vid2WAM improves novel-task generalization and data efficiency under limited expert demonstrations while preserving low-latency inference.

Problem

• World Action Models
• expert demonstrations
• generalization
• scalability
• robot policy learning

Innovation

• World Action Models
• video diffusion priors
• offline distillation
• inverse dynamics
• residual action adaptation