From Image- to Pixel-level: Label-efficient Hyperspectral Image Reconstruction

2025-03-10

AI Summary

In hyperspectral image reconstruction, image-level supervision incurs high annotation costs and low efficiency, whereas point-spectrum acquisition—though efficient—suffers from poor cross-scene generalization and insufficient spectral information utilization. To address these limitations, we propose Pixel-SSR, a novel pixel-level spectral super-resolution paradigm. Pixel-SSR takes an RGB image and sparse point spectra as input, introduces a Gamma-distribution-based strategy for synthesizing realistic point spectra, and designs a Dynamic Prompt Mamba (DyPro-Mamba) architecture with three parallel branches to jointly model spatial distribution, edge structure, and spectral dependencies. By integrating RGB-spectral multimodal features and adopting pixel-level supervision, Pixel-SSR achieves state-of-the-art reconstruction accuracy—comparable to image-level supervised methods—while requiring orders of magnitude fewer point-spectrum annotations. It significantly outperforms both image-level supervised and unsupervised approaches in comprehensive horizontal and vertical evaluations.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Current hyperspectral image (HSI) reconstruction methods primarily rely on image-level approaches, which are time-consuming to form abundant high-quality HSIs through imagers. In contrast, spectrometers offer a more efficient alternative by capturing high-fidelity point spectra, enabling pixel-level HSI reconstruction that balances accuracy and label efficiency. To this end, we introduce a pixel-level spectral super-resolution (Pixel-SSR) paradigm that reconstructs HSI from RGB and point spectra. Despite its advantages, Pixel-SSR presents two key challenges: 1) generalizability to novel scenes lacking point spectra, and 2) effective information extraction to promote reconstruction accuracy. To address the first challenge, a Gamma-modeled strategy is investigated to synthesize point spectra based on their intrinsic properties, including nonnegativity, a skewed distribution, and a positive correlation. Furthermore, complementary three-branch prompts from RGB and point spectra are extracted with a Dynamic Prompt Mamba (DyPro-Mamba), which progressively directs the reconstruction with global spatial distributions, edge details, and spectral dependency. Comprehensive evaluations, including horizontal comparisons with leading methods and vertical assessments across unsupervised and image-level supervised paradigms, demonstrate that ours achieves competitive reconstruction accuracy with efficient label consumption.

Problem

• Pixel-level hyperspectral image reconstruction from RGB and point spectra.
• Challenges in generalizability to novel scenes without point spectra.
• Effective information extraction for accurate HSI reconstruction.

Innovation

• Pixel-level spectral super-resolution (Pixel-SSR) paradigm
• Gamma-modeled strategy for point spectra synthesis
• Dynamic Prompt Mamba (DyPro-Mamba) for information extraction