SSFT: A Lightweight Spectral-Spatial Fusion Transformer for Generic Hyperspectral Classification

2026-04-17

AI Summary

This work addresses the challenges of hyperspectral image classification—particularly high dimensionality, spectral redundancy, scarce annotations, and domain shift—which are exacerbated in non-remote-sensing scenarios by data sparsity and class imbalance. To tackle these issues, the authors propose a lightweight Spectral-Spatial Fusion Transformer (SSFT), which, for the first time, efficiently integrates decoupled spectral and spatial modeling through a cross-attention mechanism. Remarkably, SSFT achieves robust training without data augmentation while using fewer than 2% of the parameters of previous state-of-the-art methods. The model attains top performance on the HSI-Benchmark and remains competitive on the larger-scale SpectralEarth benchmark, demonstrating an exceptional balance between computational efficiency and generalization capability.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Remote Sensing / Geospatial AI
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Hyperspectral imaging enables fine-grained recognition of materials by capturing rich spectral signatures, but learning robust classifiers is challenging due to high dimensionality, spectral redundancy, limited labeled data, and strong domain shifts. Beyond earth observation, labeled HSI data is often scarce and imbalanced, motivating compact models for generic hyperspectral classification across diverse acquisition regimes. We propose the lightweight Spectral-Spatial Fusion Transformer (SSFT), which factorizes representation learning into spectral and spatial pathways and integrates them via cross-attention to capture complementary wavelength-dependent and structural information. We evaluate our SSFT on the challenging HSI-Benchmark, a heterogeneous multi-dataset benchmark covering earth observation, fruit condition assessment, and fine-grained material recognition. SSFT achieves state-of-the-art overall performance, ranking first while using less than 2% of the parameters of the previous leading method. We further evaluate transfer to the substantially larger SpectralEarth benchmark under the official protocol, where SSFT remains competitive despite its compact size. Ablation studies show that both spectral and spatial pathways are crucial, with spatial modeling contributing most, and that SSFT remains robust without data augmentation.

Problem

• hyperspectral classification
• high dimensionality
• limited labeled data
• domain shift
• data scarcity

Innovation

• Spectral-Spatial Fusion
• Lightweight Transformer
• Cross-Attention
• Generic Hyperspectral Classification
• Parameter Efficiency