Mixed-View Panorama Synthesis using Geospatially Guided Diffusion

2024-07-12

arXiv.org

AI Summary

This paper introduces hybrid-view panoramic image synthesis: generating high-fidelity novel panoramas given only a few street-level panoramas of a target region and their corresponding satellite imagery. This setting bridges the limitations of conventional single-view methods (relying solely on panoramas) and pure cross-view approaches (using only satellite images), enabling better generalization to arbitrary global locations. To address the extreme sparsity of geographically distributed panoramas, we propose a geospatial guidance mechanism and develop a diffusion-based multimodal fusion framework. It jointly models geometric and semantic correspondences between spherical panoramas and planar satellite images via attention mechanisms. Experiments demonstrate that our method maintains stable, high-quality generation even under severe panorama scarcity or when source panoramas are geographically distant from the target location—significantly outperforming existing single-view and cross-view baselines.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We introduce the task of mixed-view panorama synthesis, where the goal is to synthesize a novel panorama given a small set of input panoramas and a satellite image of the area. This contrasts with previous work which only uses input panoramas (same-view synthesis), or an input satellite image (cross-view synthesis). We argue that the mixed-view setting is the most natural to support panorama synthesis for arbitrary locations worldwide. A critical challenge is that the spatial coverage of panoramas is uneven, with few panoramas available in many regions of the world. We introduce an approach that utilizes diffusion-based modeling and an attention-based architecture for extracting information from all available input imagery. Experimental results demonstrate the effectiveness of our proposed method. In particular, our model can handle scenarios when the available panoramas are sparse or far from the location of the panorama we are attempting to synthesize. The project page is available at https://mixed-view.github.io

Problem

• Synthesize novel panoramas using mixed-view inputs
• Address uneven spatial coverage of panorama data
• Handle sparse or distant panorama inputs effectively

Innovation

• Geospatially guided diffusion for panorama synthesis
• Attention-based architecture for input extraction
• Handles sparse or distant panorama inputs