Hyperspectral Adapter for Object Tracking based on Hyperspectral Video

2025-03-28

AI Summary

To address spectral information loss and inefficient full-network fine-tuning in hyperspectral video object tracking, this paper proposes a lightweight and efficient transfer learning framework. We introduce the Hyperspectral Adapter (HAS/HAM), the first adapter specifically designed for hyperspectral tracking, built upon a Transformer architecture with dual-path self-attention and MLP modules; it enables adaptation of RGB-pretrained trackers via fine-tuning only a small number of parameters. Additionally, we propose a Hyperspectral Enhancement Input (HEI) mechanism to explicitly preserve raw multi-band spectral features, avoiding distortion from conventional spectral dimensionality reduction. Our method achieves state-of-the-art performance across four multispectral benchmarks, demonstrating superior robustness in complex scenarios and significantly improved deployment efficiency.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Object tracking based on hyperspectral video attracts increasing attention to the rich material and motion information in the hyperspectral videos. The prevailing hyperspectral methods adapt pretrained RGB-based object tracking networks for hyperspectral tasks by fine-tuning the entire network on hyperspectral datasets, which achieves impressive results in challenging scenarios. However, the performance of hyperspectral trackers is limited by the loss of spectral information during the transformation, and fine-tuning the entire pretrained network is inefficient for practical applications. To address the issues, a new hyperspectral object tracking method, hyperspectral adapter for tracking (HyA-T), is proposed in this work. The hyperspectral adapter for the self-attention (HAS) and the hyperspectral adapter for the multilayer perceptron (HAM) are proposed to generate the adaption information and to transfer the multi-head self-attention (MSA) module and the multilayer perceptron (MLP) in pretrained network for the hyperspectral object tracking task by augmenting the adaption information into the calculation of the MSA and MLP. Additionally, the hyperspectral enhancement of input (HEI) is proposed to augment the original spectral information into the input of the tracking network. The proposed methods extract spectral information directly from the hyperspectral images, which prevent the loss of the spectral information. Moreover, only the parameters in the proposed methods are fine-tuned, which is more efficient than the existing methods. Extensive experiments were conducted on four datasets with various spectral bands, verifing the effectiveness of the proposed methods. The HyA-T achieves state-of-the-art performance on all the datasets.

Problem

• Enhancing object tracking using hyperspectral video data
• Reducing spectral information loss in tracking networks
• Improving efficiency of hyperspectral tracker adaptation

Innovation

• HyA-T method adapts pretrained networks efficiently
• HAS and HAM modules preserve spectral information
• HEI enhances input with original spectral data