PanoDreamer: 3D Panorama Synthesis from a Single Image

2024-12-06

arXiv.org

AI Summary

This work addresses the challenging problem of generating geometrically consistent and visually faithful 360° 3D scenes from a single 2D image. We propose an optimization-driven framework that jointly estimates omnidirectional panoramas and dense depth maps. Our key contribution is the first formulation of single-image 360° panorama generation and depth estimation as a coupled bi-objective optimization problem, solved efficiently via an alternating minimization algorithm. To ensure geometric validity and texture coherence, we introduce a depth-guided spherical projection operator and a novel hole-filling mechanism leveraging 3D spherical geometry. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple benchmarks in three critical metrics: panoramic consistency, depth accuracy, and novel-view rendering quality. Unlike prior approaches, our end-to-end framework enables high-fidelity, reconstructable 360° 3D scene synthesis without requiring multi-view inputs or explicit 3D supervision.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Constraint Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

In this paper, we present PanoDreamer, a novel method for producing a coherent 360$^circ$ 3D scene from a single input image. Unlike existing methods that generate the scene sequentially, we frame the problem as single-image panorama and depth estimation. Once the coherent panoramic image and its corresponding depth are obtained, the scene can be reconstructed by inpainting the small occluded regions and projecting them into 3D space. Our key contribution is formulating single-image panorama and depth estimation as two optimization tasks and introducing alternating minimization strategies to effectively solve their objectives. We demonstrate that our approach outperforms existing techniques in single-image 360$^circ$ scene reconstruction in terms of consistency and overall quality.

Problem

• Generates 360° 3D scene from single image
• Estimates panorama and depth via optimization
• Improves consistency and quality in reconstruction

Innovation

• Single-image panorama and depth estimation optimization
• Alternating minimization for coherent 360° 3D reconstruction
• Inpainting and projecting occluded regions into 3D space