ViewFusion: Learning Composable Diffusion Models for Novel View Synthesis

2024-02-05

arXiv.org

AI Summary

Existing novel view synthesis methods—such as NeRF and end-to-end models—suffer from limited input flexibility, pose dependency, and poor generalization. To address these limitations, we propose ViewFusion, an end-to-end composable diffusion model for multi-view image synthesis. Our approach requires no camera pose supervision and accepts arbitrary numbers of uncalibrated input views. It introduces joint multi-view denoising coupled with pixel-wise dynamic weight masks, enabling adaptive, region-aware noise gradient weighting and fusion. ViewFusion exhibits strong cross-scene and cross-category generalization, supports variable numbers of input views at both training and inference time, and maintains generation plausibility even under highly underdetermined conditions. Evaluated on the NMR dataset, it achieves state-of-the-art performance. However, it does not produce explicit 3D representations, and its inference speed is inherently constrained by the iterative diffusion sampling process.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Deep learning is providing a wealth of new approaches to the old problem of novel view synthesis, from Neural Radiance Field (NeRF) based approaches to end-to-end style architectures. Each approach offers specific strengths but also comes with specific limitations in their applicability. This work introduces ViewFusion, a state-of-the-art end-to-end generative approach to novel view synthesis with unparalleled flexibility. ViewFusion consists in simultaneously applying a diffusion denoising step to any number of input views of a scene, then combining the noise gradients obtained for each view with an (inferred) pixel-weighting mask, ensuring that for each region of the target scene only the most informative input views are taken into account. Our approach resolves several limitations of previous approaches by (1) being trainable and generalizing across multiple scenes and object classes, (2) adaptively taking in a variable number of pose-free views at both train and test time, (3) generating plausible views even in severely undetermined conditions (thanks to its generative nature) -- all while generating views of quality on par or even better than state-of-the-art methods. Limitations include not generating a 3D embedding of the scene, resulting in a relatively slow inference speed, and our method only being tested on the relatively small dataset NMR. Code is available.

Problem

• Novel view synthesis with unparalleled flexibility
• Adaptive handling of variable input views
• Generating plausible views in underdetermined conditions

Innovation

• Uses diffusion denoising for multi-view synthesis
• Combines noise gradients with pixel-weighting masks
• Adapts to variable pose-free views flexibly