WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories

2026-03-02

AI Summary

Existing video diffusion models struggle to generate multi-view consistent 3D scenes due to weak camera control and view inconsistency. To address this, this work proposes a dual geometric memory mechanism: a global geometric memory module injects 3D structural priors through incrementally updated point clouds, while a spatial stereo memory module—based on 3D correspondences—constrains the attention mechanism to enforce geometric consistency. Without requiring joint training, the method enables high-fidelity, multi-view consistent video generation under precise camera control when built upon distilled video diffusion models, significantly improving 3D reconstruction quality. Experiments demonstrate that the approach achieves state-of-the-art performance across multiple benchmarks for camera-guided generation and 3D reconstruction, offering both flexible controllability and strong generalization capabilities.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Recent advances in foundational Video Diffusion Models (VDMs) have yielded significant progress. Yet, despite the remarkable visual quality of generated videos, reconstructing consistent 3D scenes from these outputs remains challenging, due to limited camera controllability and inconsistent generated content when viewed from distinct camera trajectories. In this paper, we propose WorldStereo, a novel framework that bridges camera-guided video generation and 3D reconstruction via two dedicated geometric memory modules. Formally, the global-geometric memory enables precise camera control while injecting coarse structural priors through incrementally updated point clouds. Moreover, the spatial-stereo memory constrains the model's attention receptive fields with 3D correspondence to focus on fine-grained details from the memory bank. These components enable WorldStereo to generate multi-view-consistent videos under precise camera control, facilitating high-quality 3D reconstruction. Furthermore, the flexible control branch-based WorldStereo shows impressive efficiency, benefiting from the distribution matching distilled VDM backbone without joint training. Extensive experiments across both camera-guided video generation and 3D reconstruction benchmarks demonstrate the effectiveness of our approach. Notably, we show that WorldStereo acts as a powerful world model, tackling diverse scene generation tasks (whether starting from perspective or panoramic images) with high-fidelity 3D results. Models will be released.

Problem

• 3D reconstruction
• video generation
• camera control
• multi-view consistency
• scene consistency

Innovation

• WorldStereo
• 3D geometric memories
• camera-guided video generation
• multi-view consistency
• 3D reconstruction