HDRT: Infrared Capture for HDR Imaging

2024-06-08

arXiv.org

AI Summary

To address detail loss in under-/over-exposed regions of standard dynamic range (SDR) images and the susceptibility of conventional HDR methods to ghosting and incomplete data, this paper proposes HDRT—the first single-exposure HDR reconstruction framework integrating thermal infrared (IR) and visible-light modalities. Methodologically, we introduce the first paired HDR-thermal dataset and design HDRTNet, a dual-branch shallow feature fusion network incorporating IR-RGB cross-modal alignment and inverse tone-mapping enhancement modules. Quantitatively, HDRT achieves significant PSNR/SSIM improvements over state-of-the-art methods in both under- and over-exposed scenarios; qualitatively, it delivers superior visual fidelity and robustness across diverse illumination conditions. Key contributions include: (i) establishing the first thermal–visible collaborative HDR acquisition paradigm; (ii) releasing the first thermal-aware HDR benchmark dataset; and (iii) proposing a lightweight, efficient dual-modal feature fusion architecture.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

Capturing real world lighting is a long standing challenge in imaging and most practical methods acquire High Dynamic Range (HDR) images by either fusing multiple exposures, or boosting the dynamic range of Standard Dynamic Range (SDR) images. Multiple exposure capture is problematic as it requires longer capture times which can often lead to ghosting problems. The main alternative, inverse tone mapping is an ill-defined problem that is especially challenging as single captured exposures usually contain clipped and quantized values, and are therefore missing substantial amounts of content. To alleviate this, we propose a new approach, High Dynamic Range Thermal (HDRT), for HDR acquisition using a separate, commonly available, thermal infrared (IR) sensor. We propose a novel deep neural method (HDRTNet) which combines IR and SDR content to generate HDR images. HDRTNet learns to exploit IR features linked to the RGB image and the IR-specific parameters are subsequently used in a dual branch method that fuses features at shallow layers. This produces an HDR image that is significantly superior to that generated using naive fusion approaches. To validate our method, we have created the first HDR and thermal dataset, and performed extensive experiments comparing HDRTNet with the state-of-the-art. We show substantial quantitative and qualitative quality improvements on both over- and under-exposed images, showing that our approach is robust to capturing in multiple different lighting conditions.

Problem

• Overcome SDR image limitations in underexposed or overexposed regions.
• Address ghosting and incomplete data in traditional HDR methods.
• Introduce IR-guided HDR imaging with a novel dataset and method.

Innovation

• First comprehensive HDR and thermal IR dataset
• HDRTNet: deep neural fusion of IR and SDR
• 50,000 images across diverse lighting conditions