Neural Augmentation Based Panoramic High Dynamic Range Stitching

2024-09-07

Neurocomputing

AI Summary

Existing panoramic HDR stitching from multi-exposure LDR images suffers from saturation artifacts, photometric inconsistencies, and ghosting. To address these issues, this paper proposes a synergistic framework combining physics-driven initial estimation and data-driven refinement, guided by the overlapping field of view (OFOV). The method integrates a physical imaging model with a neural image enhancement network to construct exposure-consistent panoramic LDR sequences, followed by multi-scale exposure fusion and geometrically synchronized stitching for HDR panorama synthesis. Crucially, OFOV is introduced for the first time as both a physical constraint and a learning guidance signal, unifying the modeling of exposure discrepancies and spatial alignment. Extensive experiments on multiple HDR scenes demonstrate that our approach significantly outperforms state-of-the-art panoramic stitching methods, achieving superior performance in suppressing saturation, extending dynamic range, preserving fine details, and eliminating ghosting artifacts.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Due to saturated regions of inputting low dynamic range (LDR) images and large intensity changes among the LDR images caused by different exposures, it is challenging to produce an information enriched panoramic LDR image without visual artifacts for a high dynamic range (HDR) scene through stitching multiple geometrically synchronized LDR images with different exposures and pairwise overlapping fields of views (OFOVs). Fortunately, the stitching of such images is innately a perfect scenario for the fusion of a physics-driven approach and a data-driven approach due to their OFOVs. Based on this new insight, a novel neural augmentation based panoramic HDR stitching algorithm is proposed in this paper. The physics-driven approach is built up using the OFOVs. Different exposed images of each view are initially generated by using the physics-driven approach, are then refined by a data-driven approach, and are finally used to produce panoramic LDR images with different exposures. All the panoramic LDR images with different exposures are combined together via a multi-scale exposure fusion algorithm to produce the final panoramic LDR image. Experimental results demonstrate the proposed algorithm outperforms existing panoramic stitching algorithms.

Problem

• Neural augmentation for HDR stitching
• Overcoming LDR image saturation issues
• Fusing physics and data-driven approaches

Innovation

• Neural augmentation enhances HDR stitching.
• Physics-driven and data-driven methods fused.
• Multi-scale exposure fusion creates panoramic LDR.