Multispectral to Hyperspectral using Pretrained Foundational model

2025-02-26

AI Summary

Low spectral resolution of multispectral remote sensing data impedes accurate retrieval of greenhouse gases (e.g., CH₄ and NO₂). To address this, we propose a dual-path Transformer model—integrating spectral and spatial-spectral branches—for cross-modal super-resolution reconstruction from wide-coverage multispectral imagery (e.g., Sentinel-2, HLS) to hyperspectral imagery. Our approach innovatively adapts pretrained foundation models to spectral super-resolution, jointly modeling spectral response functions and spatial-spectral correlations via a Spectral Transformer and a Spatial-Spectral Transformer. We conduct two-stage training: pretraining on EnMAP and EMIT hyperspectral data, followed by paired fine-tuning (Sentinel-2/EnMAP and HLS-S30/EMIT). Experiments demonstrate substantial improvements in CH₄ and NO₂ retrieval accuracy. The method faithfully reconstructs critical absorption features while preserving the temporal and spatial coverage advantages of multispectral data, enabling high-frequency, large-scale atmospheric composition monitoring.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance

Abstract

Hyperspectral imaging provides detailed spectral information, offering significant potential for monitoring greenhouse gases like CH4 and NO2. However, its application is constrained by limited spatial coverage and infrequent revisit times. In contrast, multispectral imaging delivers broader spatial and temporal coverage but lacks the spectral granularity required for precise GHG detection. To address these challenges, this study proposes Spectral and Spatial-Spectral transformer models that reconstruct hyperspectral data from multispectral inputs. The models in this paper are pretrained on EnMAP and EMIT datasets and fine-tuned on spatio-temporally aligned (Sentinel-2, EnMAP) and (HLS-S30, EMIT) image pairs respectively. Our model has the potential to enhance atmospheric monitoring by combining the strengths of hyperspectral and multispectral imaging systems.

Problem

• Reconstruct hyperspectral data from multispectral inputs
• Enhance atmospheric monitoring capabilities
• Combine strengths of hyperspectral and multispectral imaging

Innovation

• Transforms multispectral to hyperspectral data
• Utilizes pretrained foundational transformer models
• Enhances atmospheric monitoring precision