ReplicateAnyScene: Zero-Shot Video-to-3D Composition via Textual-Visual-Spatial Alignment

2026-04-12

AI Summary

Existing approaches struggle to achieve zero-shot, fully automatic reconstruction of semantically consistent and physically plausible compositional 3D scenes from ordinary videos, often relying on manual prompts or additional inputs and being constrained by training biases. This work proposes a five-stage cascaded framework that, for the first time, enables video-to-compositional-3D-scene reconstruction without any human intervention or auxiliary data. By integrating vision foundation models, the method aligns general-purpose priors across textual, visual, and spatial dimensions in a structured manner and leverages structured 3D representation learning. To facilitate standardized evaluation, we introduce the C3DR benchmark. Experiments demonstrate that our approach substantially outperforms existing methods, generating high-quality 3D scenes that are both semantically coherent and physically reasonable.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Humans exhibit an innate capacity to rapidly perceive and segment objects from video observations, and even mentally assemble them into structured 3D scenes. Replicating such capability, termed compositional 3D reconstruction, is pivotal for the advancement of Spatial Intelligence and Embodied AI. However, existing methods struggle to achieve practical deployment due to the insufficient integration of cross-modal information, leaving them dependent on manual object prompting, reliant on auxiliary visual inputs, and restricted to overly simplistic scenes by training biases. To address these limitations, we propose ReplicateAnyScene, a framework capable of fully automated and zero-shot transformation of casually captured videos into compositional 3D scenes. Specifically, our pipeline incorporates a five-stage cascade to extract and structurally align generic priors from vision foundation models across textual, visual, and spatial dimensions, grounding them into structured 3D representations and ensuring semantic coherence and physical plausibility of the constructed scenes. To facilitate a more comprehensive evaluation of this task, we further introduce the C3DR benchmark to assess reconstruction quality from diverse aspects. Extensive experiments demonstrate the superiority of our method over existing baselines in generating high-quality compositional 3D scenes.

Problem

• compositional 3D reconstruction
• zero-shot
• video-to-3D
• cross-modal alignment
• Spatial Intelligence

Innovation

• zero-shot
• compositional 3D reconstruction
• cross-modal alignment
• vision foundation models
• Spatial Intelligence