Reconstruct, Inpaint, Finetune: Dynamic Novel-view Synthesis from Monocular Videos

2025-07-16

AI Summary

To address geometric distortion and poor generalization in monocular dynamic video novel-view synthesis, this paper proposes CogNVS: a two-stage framework decoupling the task into geometry-driven 3D dynamic reconstruction (with differentiable rendering) and generative-prior-guided 2D video completion. We introduce the first self-supervised video diffusion model, enabling zero-shot training on unlabeled real-world videos without manual annotations. Additionally, we incorporate lightweight test-time fine-tuning to enhance cross-scene generalization. Crucially, our method avoids expensive 4D voxel representations and mitigates geometric collapse during feedforward training. Evaluated on multiple dynamic scene benchmarks, CogNVS significantly outperforms prior approaches, producing novel views with high geometric fidelity, strong temporal consistency, and superior detail recovery. The framework establishes an efficient, robust, and scalable paradigm for monocular dynamic novel-view synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We explore novel-view synthesis for dynamic scenes from monocular videos. Prior approaches rely on costly test-time optimization of 4D representations or do not preserve scene geometry when trained in a feed-forward manner. Our approach is based on three key insights: (1) covisible pixels (that are visible in both the input and target views) can be rendered by first reconstructing the dynamic 3D scene and rendering the reconstruction from the novel-views and (2) hidden pixels in novel views can be "inpainted" with feed-forward 2D video diffusion models. Notably, our video inpainting diffusion model (CogNVS) can be self-supervised from 2D videos, allowing us to train it on a large corpus of in-the-wild videos. This in turn allows for (3) CogNVS to be applied zero-shot to novel test videos via test-time finetuning. We empirically verify that CogNVS outperforms almost all prior art for novel-view synthesis of dynamic scenes from monocular videos.

Problem

• Novel-view synthesis for dynamic scenes from monocular videos
• Preserving scene geometry without costly 4D optimization
• Self-supervised inpainting of hidden pixels in novel views

Innovation

• Reconstruct dynamic 3D scenes for covisible pixels
• Inpaint hidden pixels using 2D video diffusion models
• Self-supervised training on large video datasets