Geometric Consistency Refinement for Single Image Novel View Synthesis via Test-Time Adaptation of Diffusion Models

2025-04-11

AI Summary

In single-image novel view synthesis (NVS), diffusion models generate high-fidelity images but often violate epipolar geometry constraints, leading to geometric inconsistencies. To address this, we propose a plug-and-play, fine-tuning-free method that jointly optimizes the initial sampling noise at test time. Our approach employs differentiable image-matching and epipolar geometry loss functions to simultaneously enforce both photorealism and geometric consistency under the target camera pose. Crucially, this is the first work to explicitly incorporate epipolar constraints into the diffusion sampling process. Evaluated on the MegaScenes dataset, our method reduces reprojection error by 32% while significantly improving geometric consistency—without compromising visual quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Non-convex Optimization

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Diffusion models for single image novel view synthesis (NVS) can generate highly realistic and plausible images, but they are limited in the geometric consistency to the given relative poses. The generated images often show significant errors with respect to the epipolar constraints that should be fulfilled, as given by the target pose. In this paper we address this issue by proposing a methodology to improve the geometric correctness of images generated by a diffusion model for single image NVS. We formulate a loss function based on image matching and epipolar constraints, and optimize the starting noise in a diffusion sampling process such that the generated image should both be a realistic image and fulfill geometric constraints derived from the given target pose. Our method does not require training data or fine-tuning of the diffusion models, and we show that we can apply it to multiple state-of-the-art models for single image NVS. The method is evaluated on the MegaScenes dataset and we show that geometric consistency is improved compared to the baseline models while retaining the quality of the generated images.

Problem

• Improving geometric consistency in single image novel view synthesis
• Addressing epipolar constraint errors in diffusion-generated NVS images
• Optimizing diffusion sampling noise for realistic and geometrically correct outputs

Innovation

• Optimizes starting noise for geometric consistency
• Uses epipolar constraints in loss function
• Applies test-time adaptation without retraining