RealWonder: Real-Time Physical Action-Conditioned Video Generation

2026-03-05

AI Summary

Existing video generation models struggle to simulate the physical consequences of 3D actions—such as applied forces or robotic manipulation—due to their neglect of the dynamic impact such actions exert on scene structure. This work proposes the first physics-simulation-based, real-time action-conditioned video generation framework. It initializes a 3D scene from a single input image, leverages a physics engine to simulate responses of rigid bodies, deformable objects, fluids, and granular materials, and encodes the simulation outcomes into a joint optical flow and RGB representation. A four-stage distilled diffusion model then synthesizes high-fidelity videos from this representation. The method achieves real-time interaction at 13.2 FPS at 480×832 resolution, supporting diverse inputs including forces, robotic actions, and camera control. Code and models are publicly released.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Web Mining and Content Analysis: Web data generation and simulation
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Current video generation models cannot simulate physical consequences of 3D actions like forces and robotic manipulations, as they lack structural understanding of how actions affect 3D scenes. We present RealWonder, the first real-time system for action-conditioned video generation from a single image. Our key insight is using physics simulation as an intermediate bridge: instead of directly encoding continuous actions, we translate them through physics simulation into visual representations (optical flow and RGB) that video models can process. RealWonder integrates three components: 3D reconstruction from single images, physics simulation, and a distilled video generator requiring only 4 diffusion steps. Our system achieves 13.2 FPS at 480x832 resolution, enabling interactive exploration of forces, robot actions, and camera controls on rigid objects, deformable bodies, fluids, and granular materials. We envision RealWonder opens new opportunities to apply video models in immersive experiences, AR/VR, and robot learning. Our code and model weights are publicly available in our project website: https://liuwei283.github.io/RealWonder/

Problem

• video generation
• physical simulation
• action-conditioned
• 3D scene understanding
• real-time

Innovation

• physics simulation
• action-conditioned video generation
• real-time video synthesis
• diffusion distillation
• 3D scene interaction