Generative Model-Assisted Demosaicing for Cross-multispectral Cameras

2025-03-04

AI Summary

This work addresses three key challenges in spectral demosaicing for spectral filter array (SFA)-based multispectral imaging: (1) poor generalization of simulation-trained models due to absence of real ground-truth labels; (2) substantial inter-camera spectral response variations; and (3) visual artifacts arising from naive interpolation in hard regions. We propose a generative model-assisted hybrid supervision framework comprising three stages: self-supervised generative pretraining, frequency-domain hard-patch selection-driven pseudo-pair generation, and real-data-targeted fine-tuning. We introduce the first frequency-domain pseudo-label construction and hard-region identification mechanism, and establish UniSpecTest—the first real-world multispectral mosaic benchmark. Our method achieves significant gains over state-of-the-art methods on both synthetic and real data. Ablation studies confirm the efficacy of each component, while cross-camera transferability and visual fidelity are substantially improved.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

As a crucial part of the spectral filter array (SFA)-based multispectral imaging process, spectral demosaicing has exploded with the proliferation of deep learning techniques. However, (1) bothering by the difficulty of capturing corresponding labels for real data or simulating the practical spectral imaging process, end-to-end networks trained in a supervised manner using simulated data often perform poorly on real data. (2) cross-camera spectral discrepancies make it difficult to apply pre-trained models to new cameras. (3) existing demosaicing networks are prone to introducing visual artifacts on hard cases due to the interpolation of unknown values. To address these issues, we propose a hybrid supervised training method with the assistance of the self-supervised generative model, which performs well on real data across different spectral cameras. Specifically, our approach consists of three steps: (1) Pre-Training step: training the end-to-end neural network on a large amount of simulated data; (2) Pseudo-Pairing step: generating pseudo-labels of real target data using the self-supervised generative model; (3) Fine-Tuning step: fine-tuning the pre-trained model on the pseudo data pairs obtained in (2). To alleviate artifacts, we propose a frequency-domain hard patch selection method that identifies artifact-prone regions by analyzing spectral discrepancies using Fourier transform and filtering techniques, allowing targeted fine-tuning to enhance demosaicing performance. Finally, we propose UniSpecTest, a real-world multispectral mosaic image dataset for testing. Ablation experiments have demonstrated the effectiveness of each training step, and extensive experiments on both synthetic and real datasets show that our method achieves significant performance gains compared to state-of-the-art techniques.

Problem

• Difficulty in capturing real data labels for supervised training.
• Cross-camera spectral discrepancies hinder model generalization.
• Existing demosaicing networks introduce visual artifacts in hard cases.

Innovation

• Hybrid supervised training with generative model
• Frequency-domain hard patch selection method
• UniSpecTest dataset for real-world testing