Modulate and Reconstruct: Learning Hyperspectral Imaging from Misaligned Smartphone Views

2025-07-02

AI Summary

Hyperspectral image reconstruction (HSR) from a single RGB image is inherently ill-posed due to severe spectral information loss. To address this limitation, we propose the first multi-view smartphone-based HSR paradigm: leveraging a triple-camera system equipped with customized spectral filters to capture complementary spectral cues. Our end-to-end network jointly models spectral filtering priors, non-rigid multi-view alignment, and deep representations to directly reconstruct hyperspectral cubes from unaligned multi-view RGB images. We introduce Doomer, the first benchmark dataset specifically designed for this task. Evaluated on our newly established benchmark, our method achieves a 30% improvement in spectral reconstruction accuracy over state-of-the-art methods. This work provides the first empirical evidence that consumer-grade smartphones—without specialized hardware—can support high-accuracy, low-cost multi-view spectral imaging, significantly advancing the practical deployment of hyperspectral technology.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Hyperspectral reconstruction (HSR) from RGB images is a fundamentally ill-posed problem due to severe spectral information loss. Existing approaches typically rely on a single RGB image, limiting reconstruction accuracy. In this work, we propose a novel multi-image-to-hyperspectral reconstruction (MI-HSR) framework that leverages a triple-camera smartphone system, where two lenses are equipped with carefully selected spectral filters. Our configuration, grounded in theoretical and empirical analysis, enables richer and more diverse spectral observations than conventional single-camera setups. To support this new paradigm, we introduce Doomer, the first dataset for MI-HSR, comprising aligned images from three smartphone cameras and a hyperspectral reference camera across diverse scenes. We show that the proposed HSR model achieves consistent improvements over existing methods on the newly proposed benchmark. In a nutshell, our setup allows 30% towards more accurately estimated spectra compared to an ordinary RGB camera. Our findings suggest that multi-view spectral filtering with commodity hardware can unlock more accurate and practical hyperspectral imaging solutions.

Problem

• Reconstruct hyperspectral images from misaligned smartphone RGB views
• Overcome spectral information loss in single RGB image HSR
• Enable accurate HSR using multi-camera smartphone setups with filters

Innovation

• Multi-image-to-hyperspectral reconstruction framework
• Triple-camera smartphone with spectral filters
• First MI-HSR dataset named Doomer