Hyperspectral image fusion, Unsupervised hyperspectral super-resolution, Modality decoupling, Self-supervised learning

2024-12-06

AI Summary

In unsupervised hyperspectral image fusion, existing methods suffer from incomplete perception of modality-complementary information and insufficient modeling of cross-modal correlations. To address these issues, this work first identifies the critical role of modality disentanglement in fusion performance and proposes MossFuse—a novel end-to-end self-supervised framework. MossFuse introduces a modality-disentangled architecture that explicitly separates shared and complementary features via a subspace clustering loss, and enhances multimodal synergy through joint spatial-spectral representation aggregation. Evaluated on multiple benchmark datasets, MossFuse consistently outperforms state-of-the-art unsupervised methods in hyperspectral super-resolution reconstruction, achieving superior accuracy with significantly fewer parameters and faster inference speed—demonstrating both high performance and lightweight efficiency.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Hyperspectral and Multispectral Image Fusion (HMIF) aims to fuse low-resolution hyperspectral images (LR-HSIs) and high-resolution multispectral images (HR-MSIs) to reconstruct high spatial and high spectral resolution images. Current methods typically apply direct fusion from the two modalities without effective supervision, leading to an incomplete perception of deep modality-complementary information and a limited understanding of inter-modality correlations. To address these issues, we propose a simple yet effective solution for unsupervised HMIF, revealing that modality decoupling is key to improving fusion performance. Specifically, we propose an end-to-end self-supervised extbf{Mo}dality-Decoupled extbf{S}patial- extbf{S}pectral Fusion ( extbf{MossFuse}) framework that decouples shared and complementary information across modalities and aggregates a concise representation of both LR-HSIs and HR-MSIs to reduce modality redundancy. Also, we introduce the subspace clustering loss as a clear guide to decouple modality-shared features from modality-complementary ones. Systematic experiments over multiple datasets demonstrate that our simple and effective approach consistently outperforms the existing HMIF methods while requiring considerably fewer parameters with reduced inference time. The anonymous source code is in href{https://github.com/dusongcheng/MossFuse}{MossFuse}.

Problem

• Fuse LR-HSIs and HR-MSIs for high-resolution images
• Decouple shared and complementary information across modalities
• Improve unsupervised HMIF with self-supervised learning

Innovation

• Modality decoupling for improved fusion
• Self-supervised learning framework MossFuse
• Subspace clustering loss for feature decoupling