Can Video Diffusion Model Reconstruct 4D Geometry?

2025-03-27

AI Summary

This paper addresses the challenging problem of reconstructing dynamic 3D scenes (i.e., 4D geometry) from monocular video. We propose Sora3R, the first framework to transfer spatiotemporal latent priors from large-scale video diffusion models to 4D geometric reconstruction. Sora3R generates temporally consistent 4D point maps via a two-stage feed-forward pipeline—without relying on auxiliary supervision (e.g., depth, optical flow, or segmentation), iterative optimization, or global trajectory alignment. Instead, it operates solely on monocular video input. Key technical components include: (1) adaptation of a pointmap variational autoencoder (Pointmap VAE), (2) fine-tuning of a joint video–pointmap latent-space diffusion model, and (3) end-to-end inference. Extensive experiments demonstrate stable recovery of camera poses and high-fidelity 4D geometry across diverse dynamic scenes, achieving state-of-the-art performance.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Reconstructing dynamic 3D scenes (i.e., 4D geometry) from monocular video is an important yet challenging problem. Conventional multiview geometry-based approaches often struggle with dynamic motion, whereas recent learning-based methods either require specialized 4D representation or sophisticated optimization. In this paper, we present Sora3R, a novel framework that taps into the rich spatiotemporal priors of large-scale video diffusion models to directly infer 4D pointmaps from casual videos. Sora3R follows a two-stage pipeline: (1) we adapt a pointmap VAE from a pretrained video VAE, ensuring compatibility between the geometry and video latent spaces; (2) we finetune a diffusion backbone in combined video and pointmap latent space to generate coherent 4D pointmaps for every frame. Sora3R operates in a fully feedforward manner, requiring no external modules (e.g., depth, optical flow, or segmentation) or iterative global alignment. Extensive experiments demonstrate that Sora3R reliably recovers both camera poses and detailed scene geometry, achieving performance on par with state-of-the-art methods for dynamic 4D reconstruction across diverse scenarios.

Problem

• Reconstructing 4D geometry from monocular video
• Overcoming dynamic motion challenges in 3D scenes
• Inferring 4D pointmaps without iterative alignment

Innovation

• Uses video diffusion models for 4D reconstruction
• Two-stage pipeline with pointmap VAE adaptation
• Feedforward processing without external modules