NIR-Assisted Image Denoising: A Selective Fusion Approach and A Real-World Benchmark Dataset

2024-04-12

IEEE transactions on multimedia

AI Summary

To address the challenge of jointly suppressing noise and preserving fine details in RGB image denoising under extremely low-light conditions, this paper proposes a near-infrared (NIR)-assisted selective fusion framework. To overcome cross-modal content inconsistency between NIR and RGB and the scarcity of real-world paired data, we design a global-local collaborative modulation Selective Fusion Module (SFM), enabling plug-and-play, efficient fusion of deep cross-modal features. We further introduce Real-NAID—the first large-scale, real-scenario NIR-RGB paired denoising benchmark dataset covering multiple noise levels. Our method adopts end-to-end supervised learning with realistic noise modeling. Extensive experiments demonstrate that our approach achieves significant improvements over state-of-the-art methods on both synthetic and real-world benchmarks. For reproducibility and community advancement, we fully open-source the code, pre-trained models, and the Real-NAID dataset.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Despite the significant progress in image denoising, it is still challenging to restore fine-scale details while removing noise, especially in extremely low-light environments. Leveraging near-infrared (NIR) images to assist visible RGB image denoising shows the potential to address this issue, becoming a promising technology. Nonetheless, existing works still struggle with taking advantage of NIR information effectively for real-world image denoising, due to the content inconsistency between NIR-RGB images and the scarcity of real-world paired datasets. To alleviate the problem, we propose an efficient Selective Fusion Module (SFM), which can be plug-and-played into the advanced denoising networks to merge the deep NIR-RGB features. Specifically, we sequentially perform the global and local modulation for NIR and RGB features, and then integrate the two modulated features. Furthermore, we present a Real-world NIR-Assisted Image Denoising (Real-NAID) dataset, which covers diverse scenarios as well as various noise levels. Extensive experiments on both synthetic and our real-world datasets demonstrate that the proposed method achieves better results than state-of-the-art ones. The dataset, codes, and pre-trained models will be publicly available at https://github.com/ronjonxu/NAID.

Problem

• Restore fine-scale details in low-light image denoising.
• Effectively utilize NIR information for real-world denoising.
• Address scarcity of real-world NIR-RGB paired datasets.

Innovation

• Selective Fusion Module for NIR-RGB feature merging
• Global and local modulation of NIR-RGB features
• Real-world NIR-Assisted Image Denoising dataset creation