One Video, One World: Turning Monocular Video into Physical 4D Scenes

2026-06-30

AI Summary

Existing 4D reconstruction methods struggle to produce watertight, instance-separated meshes with standardized physical interfaces suitable for physics simulation. This work proposes OVOW, the first system capable of generating simulation-ready, instance-level 4D scenes from monocular video without requiring any training. OVOW leverages a vision-language model for object recognition and categorization, unifies reconstruction of both rigid and deformable objects into consistent meshes, recovers metric scale and pose through render-match-optimize iterations, and assembles scenes according to physical rules. Notably, it operates without category priors or skeletal rigging, supports direct vertex-based deformation modeling, and introduces the first structured Video-to-4D evaluation benchmark. Experiments demonstrate that OVOW outperforms baselines across geometric, layout, photometric, and semantic metrics, achieves inference speeds one to two orders of magnitude faster, and produces physically plausible and stable scenes validated through downstream simulation.

Abstract

We introduce \textbf{OVOW}, the first training-free system that reconstructs \emph{instance-level, simulation-ready} 4D mesh scenes from a single monocular video. Recent 4D reconstruction achieves impressive rendering quality, but its outputs (\eg, implicit fields, Gaussian primitives, or point clouds) lack the watertight topology, instance separation, and standardized physical interfaces required by physics simulators and embodied AI. OVOW closes this gap with a four-stage pipeline: a vision-language model discovers, labels, and motion-classifies all instances; category-aware reconstruction yields per-instance meshes for rigid objects and topology-consistent mesh sequences for deformable ones; an iterative render-match-optimize procedure recovers metric scale and 6-DoF pose trajectories; and physics-grounded assembly enforces ground contact and inter-object support. Crucially, we model all motion, rigid and non-rigid, through direct vertex deformation without category-specific priors or skeleton rigging, producing watertight mesh scenes ready for downstream physics simulation and editing. We further establish the first benchmark for \emph{structured Video-to-4D} evaluation, with metrics for geometric correctness, instance separation, and physical plausibility beyond visual fidelity; the same pipeline doubles as a scalable engine for \emph{synthesizing} paired video-to-4D simulation data for future 4D world models and embodied AI. Across two synthetic benchmarks (static and 4D), OVOW attains the best overall layout and geometry accuracy and the lowest photometric and semantic error among all baselines, and on monocular video runs one to two orders of magnitude faster than the baselines, while downstream physics simulation confirms its physical stability.

Problem

• 4D reconstruction
• monocular video
• physics simulation
• instance-level mesh
• physical plausibility

Innovation

• 4D reconstruction
• monocular video
• physics-ready meshes
• instance-level modeling
• training-free system