Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors

2026-06-29

AI Summary

This work addresses the challenge of sim-to-real transfer in robotic manipulation, which typically relies on costly real-world demonstration data. The authors propose a world-action model trained exclusively on synthetic data, integrating the Cosmos Policy video diffusion model, a highly domain-randomized simulation environment, and the AnyTask motion planning pipeline. Using only 800 synthetic demonstrations per task, the method achieves zero-shot deployment on a physical Franka robot without any real-world data. In experiments involving grasping, drawer opening, and object placement tasks, the model attains an average zero-shot success rate of 35%, marking a significant step toward practical deployment of world models in real robotic systems.

Abstract

Bridging the sim-to-real gap is a core challenge in deploying learned manipulation policies. Sim-to-real learning is attractive because it can replace expensive real robot demonstrations with scalable synthetic data, yet world-action models have not previously been shown to transfer from simulation to real robotic manipulation. We study whether a world-action model can be trained from synthetic priors and deployed zero-shot in the real world. To this end, we build upon Cosmos Policy, a video diffusion model adapted for visuomotor control. We construct simulation environments with extensive domain randomization and generate demonstrations using the AnyTask motion planning pipeline. We evaluate our approach across object lifting, drawer opening, and pick-and-place tasks using ${\sim}800$ synthetic demonstrations per task and no real demonstrations. When deployed zero-shot on a Franka Robot, our policy attains a 35\% average success rate. To our knowledge, this represents the first successful sim-to-real transfer of a world-action model for robotic manipulation.

Problem

• sim-to-real transfer
• world-action models
• robotic manipulation
• zero-shot deployment
• synthetic data

Innovation

• sim-to-real transfer
• world-action model
• video diffusion model
• domain randomization
• zero-shot deployment