ReVision: High-Quality, Low-Cost Video Generation with Explicit 3D Physics Modeling for Complex Motion and Interaction

2025-04-30

AI Summary

To address poor generation quality and severe physical distortions in video synthesis involving complex motion and object interactions, this paper proposes a three-stage physics-enhanced framework. First, a coarse video is generated using a video diffusion model. Second, an object-centric explicit 3D representation is constructed and refined via differentiable parametric physical priors to ensure physically plausible motion. Third, high-fidelity video is regenerated conditioned on the refined 3D motion trajectories. This work is the first to seamlessly integrate differentiable parametric physical modeling into pre-trained video diffusion models (e.g., Stable Video Diffusion), achieving superior physical consistency and motion controllability over larger 13B+ parameter models—despite using only 1.5B parameters. Experiments demonstrate substantial improvements over state-of-the-art methods in motion fidelity and temporal coherence, particularly in complex interaction scenarios, while reducing inference cost by over 80%.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In recent years, video generation has seen significant advancements. However, challenges still persist in generating complex motions and interactions. To address these challenges, we introduce ReVision, a plug-and-play framework that explicitly integrates parameterized 3D physical knowledge into a pretrained conditional video generation model, significantly enhancing its ability to generate high-quality videos with complex motion and interactions. Specifically, ReVision consists of three stages. First, a video diffusion model is used to generate a coarse video. Next, we extract a set of 2D and 3D features from the coarse video to construct a 3D object-centric representation, which is then refined by our proposed parameterized physical prior model to produce an accurate 3D motion sequence. Finally, this refined motion sequence is fed back into the same video diffusion model as additional conditioning, enabling the generation of motion-consistent videos, even in scenarios involving complex actions and interactions. We validate the effectiveness of our approach on Stable Video Diffusion, where ReVision significantly improves motion fidelity and coherence. Remarkably, with only 1.5B parameters, it even outperforms a state-of-the-art video generation model with over 13B parameters on complex video generation by a substantial margin. Our results suggest that, by incorporating 3D physical knowledge, even a relatively small video diffusion model can generate complex motions and interactions with greater realism and controllability, offering a promising solution for physically plausible video generation.

Problem

• Generating high-quality videos with complex motion and interactions
• Integrating 3D physics into video generation for realism
• Enhancing motion fidelity in low-cost video models

Innovation

• Integrates parameterized 3D physics into video generation
• Refines coarse video with 3D object-centric representation
• Enhances motion fidelity using physical prior feedback