Leveraging Multispectral Sensors for Color Correction in Mobile Cameras

2025-12-09

AI Summary

To address the limited color correction accuracy of mobile device cameras constrained by single-modality RGB input, this paper proposes an end-to-end jointly optimized framework that fuses high-resolution RGB with low-resolution multispectral sensor data. Unlike conventional approaches relying on hand-crafted priors or feature concatenation, our method preserves the full multispectral information flow throughout the pipeline and unifies the modeling of sensor response, spectral reconstruction, and color mapping—enabling seamless integration with state-of-the-art image architectures. Trained on a custom multispectral rendering dataset, the model achieves computational efficiency while significantly improving cross-device robustness. Experiments demonstrate a reduction of up to 50% in mean color error (ΔE₀₀) compared to the best RGB-only baseline, outperforming methods using multispectral priors alone, and exhibiting superior stability across diverse hardware spectral responses.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent advances in snapshot multispectral (MS) imaging have enabled compact, low-cost spectral sensors for consumer and mobile devices. By capturing richer spectral information than conventional RGB sensors, these systems can enhance key imaging tasks, including color correction. However, most existing methods treat the color correction pipeline in separate stages, often discarding MS data early in the process. We propose a unified, learning-based framework that (i) performs end-to-end color correction and (ii) jointly leverages data from a high-resolution RGB sensor and an auxiliary low-resolution MS sensor. Our approach integrates the full pipeline within a single model, producing coherent and color-accurate outputs. We demonstrate the flexibility and generality of our framework by refactoring two different state-of-the-art image-to-image architectures. To support training and evaluation, we construct a dedicated dataset by aggregating and repurposing publicly available spectral datasets, rendering under multiple RGB camera sensitivities. Extensive experiments show that our approach improves color accuracy and stability, reducing error by up to 50% compared to RGB-only and MS-driven baselines. Datasets, code, and models will be made available upon acceptance.

Problem

• Unifies color correction with multispectral and RGB data
• Integrates full pipeline in a single learning-based model
• Improves color accuracy by up to 50% over baselines

Innovation

• Unified learning-based framework for end-to-end color correction
• Jointly leverages high-resolution RGB and low-resolution multispectral sensors
• Integrates full pipeline within a single model for accuracy