Multispectral Demosaicing via Dual Cameras

2025-03-27

AI Summary

To address the low spatial resolution of multispectral (MS) images captured by dual-camera smartphones and the limitations of conventional demosaicing methods—reliant solely on single-modality priors—this paper proposes an RGB-prior-guided cross-modal MS demosaicing framework. First, we construct the first synchronized, paired RGB-MS dataset from dual-camera smartphone acquisitions. Second, we design a hybrid CNN-Transformer architecture that enables effective cross-modal feature alignment and multi-scale attention-based fusion, thereby transferring high-resolution spatial priors from the RGB stream to guide MS mosaic reconstruction. Our approach overcomes the intrinsic resolution bottleneck of single-camera MS reconstruction. Evaluated on our newly established benchmark, it achieves state-of-the-art performance, improving PSNR by over 2.1 dB compared to prior methods, and significantly outperforms both single-camera models and conventional interpolation-based approaches.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Multispectral (MS) images capture detailed scene information across a wide range of spectral bands, making them invaluable for applications requiring rich spectral data. Integrating MS imaging into multi camera devices, such as smartphones, has the potential to enhance both spectral applications and RGB image quality. A critical step in processing MS data is demosaicing, which reconstructs color information from the mosaic MS images captured by the camera. This paper proposes a method for MS image demosaicing specifically designed for dual-camera setups where both RGB and MS cameras capture the same scene. Our approach leverages co-captured RGB images, which typically have higher spatial fidelity, to guide the demosaicing of lower-fidelity MS images. We introduce the Dual-camera RGB-MS Dataset - a large collection of paired RGB and MS mosaiced images with ground-truth demosaiced outputs - that enables training and evaluation of our method. Experimental results demonstrate that our method achieves state-of-the-art accuracy compared to existing techniques.

Problem

• Reconstructs color from mosaic MS images using dual-camera setups
• Leverages RGB images to guide demosaicing of lower-fidelity MS data
• Introduces a dataset for training RGB-MS demosaicing methods

Innovation

• Dual-camera RGB-MS image demosaicing method
• RGB guides MS demosaicing for accuracy
• Large paired RGB-MS dataset for training