Dynamic View Synthesis as an Inverse Problem

2025-06-09

AI Summary

This work addresses dynamic novel-view synthesis from monocular video without training. Methodologically, it formulates the task as an inverse problem solved via a pre-trained video diffusion model, with two key innovations: (i) a K-order recursive noise representation that overcomes deterministic inversion failure caused by zero-terminal signal-to-noise ratio in standard denoising processes; and (ii) a stochastic latent-space modulation mechanism enabling visibility-aware occlusion completion. The technical pipeline integrates DDIM inversion, VAE latent-code alignment, recursive noise modeling, and stochastic latent sampling. Evaluated on multiple dynamic scenes, the approach achieves high-fidelity, artifact-free novel-view synthesis—without fine-tuning, auxiliary networks, or additional supervision. It significantly improves both reconstruction quality and generalization capability for monocular-video-driven dynamic view synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In this work, we address dynamic view synthesis from monocular videos as an inverse problem in a training-free setting. By redesigning the noise initialization phase of a pre-trained video diffusion model, we enable high-fidelity dynamic view synthesis without any weight updates or auxiliary modules. We begin by identifying a fundamental obstacle to deterministic inversion arising from zero-terminal signal-to-noise ratio (SNR) schedules and resolve it by introducing a novel noise representation, termed K-order Recursive Noise Representation. We derive a closed form expression for this representation, enabling precise and efficient alignment between the VAE-encoded and the DDIM inverted latents. To synthesize newly visible regions resulting from camera motion, we introduce Stochastic Latent Modulation, which performs visibility aware sampling over the latent space to complete occluded regions. Comprehensive experiments demonstrate that dynamic view synthesis can be effectively performed through structured latent manipulation in the noise initialization phase.

Problem

• Dynamic view synthesis from monocular videos as inverse problem
• High-fidelity synthesis without weight updates or auxiliary modules
• Structured latent manipulation for visibility-aware region completion

Innovation

• Redesign noise initialization in video diffusion
• Introduce K-order Recursive Noise Representation
• Use Stochastic Latent Modulation for visibility