Unregistered Spectral Image Fusion: Unmixing, Adversarial Learning, and Recoverability

2026-03-22

AI Summary

This work addresses the challenge of fusing spatially unregistered hyperspectral (HSI) and multispectral images (MSI) by proposing an unsupervised deep learning framework that jointly achieves HSI spatial super-resolution and MSI spectral super-resolution. The method integrates spectral unmixing with adversarial learning in a shared latent space, eliminating the need for image registration or ground-truth labels. Notably, it provides the first theoretical guarantee of recoverability for unregistered HSI–MSI fusion. Extensive experiments on multiple semi-real and real datasets demonstrate its effectiveness, significantly improving the reconstruction quality of both modalities compared to existing approaches.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This paper addresses the fusion of a pair of spatially unregistered hyperspectral image (HSI) and multispectral image (MSI) covering roughly overlapping regions. HSIs offer high spectral but low spatial resolution, while MSIs provide the opposite. The goal is to integrate their complementary information to enhance both HSI spatial resolution and MSI spectral resolution. While hyperspectral-multispectral fusion (HMF) has been widely studied, the unregistered setting remains challenging. Many existing methods focus solely on MSI super-resolution, leaving HSI unchanged. Supervised deep learning approaches were proposed for HSI super-resolution, but rely on accurate training data, which is often unavailable. Moreover, theoretical analyses largely address the co-registered case, leaving unregistered HMF poorly understood. In this work, an unsupervised framework is proposed to simultaneously super-resolve both MSI and HSI. The method integrates coupled spectral unmixing for MSI super-resolution with latent-space adversarial learning for HSI super-resolution. Theoretical guarantees on the recoverability of the super-resolution MSI and HSI are established under reasonable generative models -- providing, to our best knowledge, the first such insights for unregistered HMF. The approach is validated on semi-real and real HSI-MSI pairs across diverse conditions.

Problem

• Unregistered image fusion
• Hyperspectral image
• Multispectral image
• Super-resolution
• Image registration

Innovation

• unregistered image fusion
• spectral unmixing
• adversarial learning
• hyperspectral super-resolution
• recoverability analysis