Hyperspectral Mamba for Hyperspectral Object Tracking

2025-09-09

AI Summary

To address insufficient spectral information exploitation, weak temporal modeling, and lack of cross-depth feature interaction in hyperspectral object tracking, this paper proposes HyMamba—a novel state-space-based architecture. Its core contributions are threefold: (1) the Spectral State Integration (SSI) module, which explicitly incorporates the spectral dimension into state-space modeling for the first time; (2) the Hyperspectral Mamba (HSM) module, employing a tri-directional scanning mechanism to jointly capture spectral–spatial–temporal dependencies; and (3) a complementary fusion strategy integrating pseudo-color images with raw hyperspectral features for multimodal progressive optimization. Evaluated on seven benchmark datasets, HyMamba achieves state-of-the-art performance, attaining an AUC of 73.0% and DP@20 of 96.3% on HOTC2020.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: State Estimation

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Hyperspectral object tracking holds great promise due to the rich spectral information and fine-grained material distinctions in hyperspectral images, which are beneficial in challenging scenarios. While existing hyperspectral trackers have made progress by either transforming hyperspectral data into false-color images or incorporating modality fusion strategies, they often fail to capture the intrinsic spectral information, temporal dependencies, and cross-depth interactions. To address these limitations, a new hyperspectral object tracking network equipped with Mamba (HyMamba), is proposed. It unifies spectral, cross-depth, and temporal modeling through state space modules (SSMs). The core of HyMamba lies in the Spectral State Integration (SSI) module, which enables progressive refinement and propagation of spectral features with cross-depth and temporal spectral information. Embedded within each SSI, the Hyperspectral Mamba (HSM) module is introduced to learn spatial and spectral information synchronously via three directional scanning SSMs. Based on SSI and HSM, HyMamba constructs joint features from false-color and hyperspectral inputs, and enhances them through interaction with original spectral features extracted from raw hyperspectral images. Extensive experiments conducted on seven benchmark datasets demonstrate that HyMamba achieves state-of-the-art performance. For instance, it achieves 73.0% of the AUC score and 96.3% of the DP@20 score on the HOTC2020 dataset. The code will be released at https://github.com/lgao001/HyMamba.

Problem

• Capturing intrinsic spectral information in hyperspectral object tracking
• Modeling temporal dependencies and cross-depth interactions effectively
• Overcoming limitations of false-color transformation and modality fusion

Innovation

• Unifies spectral, cross-depth, temporal modeling via SSMs
• Uses Spectral State Integration module for feature refinement
• Employs directional scanning SSMs for spatial-spectral learning