World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video

2026-07-01

AI Summary

Reconstructing high-quality, freely renderable dynamic 3D scenes from monocular videos remains challenging due to occlusions, missing viewpoints, and motion blur. This work proposes a novel approach that leverages generative video models to refine and complete an initial dynamic 3D Gaussian Splatting (3DGS) reconstruction. By guiding the generative model with densely pixel-aligned renderings and distilling the synthesized content back into a unified dynamic 3DGS representation, the method achieves consistent and high-fidelity 4D reconstructions from single-view videos. It supports large viewpoint variations and complex motions, establishing a new state of the art in 4D reconstruction. The approach significantly improves novel view synthesis quality and 3D motion accuracy, and demonstrates strong generalization to real-world in-the-wild videos.

Abstract

We present World from Motion, a method for generating freely renderable dynamic 3D Gaussian representations from monocular videos. Our approach conditions a video model on dense, pixel-aligned renderings that encode appearance, geometry, and 3D scene motion along both input and target camera trajectories to correct rendering artifacts and fill in missing regions from an initial reconstruction. To train this model, we construct a dataset of aligned multiview video pairs and dynamic 3DGS representations, with simulated artifacts characteristic of monocular reconstruction. At test time, we distill the model's generations, including newly observed regions and motions, back into a single consistent, high-quality dynamic 3DGS, improving both novel-view synthesis and the underlying 3D motion. Our method sets a new state of the art in 4D reconstruction and seamlessly generalizes to in-the-wild videos with large viewpoint changes and dynamic motions.

Problem

• dynamic 3D reconstruction
• monocular video
• 4D reconstruction
• novel-view synthesis
• 3D scene motion

Innovation

• Dynamic 3D Gaussian Splatting
• Monocular Video Reconstruction
• 4D Scene Reconstruction
• Neural Rendering
• View Synthesis