Learning Transferable Dynamics Priors from Action to World Modeling

2026-06-28

AI Summary

This work proposes A2World—the first action-conditioned diffusion-based world model—designed to learn transferable dynamics priors from large-scale, multi-view robotic manipulation data. By pretraining on action-driven visual scene evolution, A2World enables long-horizon, high-fidelity simulation and joint video-action prediction, and seamlessly integrates into specialized simulators and policy learning frameworks. Experimental results demonstrate that A2World significantly improves simulation fidelity and policy prediction accuracy, effectively substituting real-world trial-and-error with efficient what-if analysis. This approach establishes a new paradigm for enhancing the generalization capabilities of robotic systems through learned world models.

Abstract

We study action-conditioned world modeling as a scalable way to learn transferable dynamics priors for robot learning. By pretraining a model to predict how actions drive visual scene evolution, the resulting world model captures reusable interaction dynamics beyond appearance-level video generation. Concretely, we pretrain a multi-view interactive base diffusion world model, A2World, on large-scale robot manipulation data with real action annotations. We validate the learned dynamics priors from two complementary perspectives. First, we adapt A2World into a task- or scene-specialized real-world simulator, A2World-sim, whose long-horizon rollouts support simulator-based policy evaluation and scalable what-if analysis by replacing real-robot rollouts with world model rollouts. Second, starting from the same pretrained weights, we adapt A2World into a video-action joint prediction model, A2World-policy, that predicts actions under visual and instruction conditioning. Experiments across simulation benchmarks and real-robot settings demonstrate that action-conditioned world model pretraining yields transferable dynamics priors that benefit both simulator-centric and policy-centric robot learning.

Problem

• world modeling
• dynamics priors
• action-conditioned
• transferable learning
• robot learning

Innovation

• action-conditioned world modeling
• transferable dynamics priors
• diffusion world model
• robot learning
• pretraining