End-to-End Unmixing with Material Prompts for Hyperspectral Object Tracking

2026-05-19

AI Summary

This work addresses the limitation of existing hyperspectral object tracking methods, which often neglect intrinsic material information or rely on external, decoupled unmixing procedures that hinder the optimization of material representations for accurate localization. To overcome this, we propose the first end-to-end trainable material-aware tracking framework that formulates object tracking as a joint optimization problem of material unmixing and spatial localization. A target-oriented weighted unmixing loss is introduced to align material representations with localization accuracy, and a dual-branch wavelet-enhanced material prompting module is designed to enable efficient spatial–material interaction in the frequency domain. Extensive experiments on standard hyperspectral tracking benchmarks demonstrate state-of-the-art performance, validating the method’s effectiveness and generalization capability.

Abstract

Hyperspectral imagery encodes rich material properties that can improve tracking robustness under appearance ambiguity, illumination change, and background clutter. However, due to the limited availability of hyperspectral video data, many existing methods adapt pretrained RGB trackers via spatial or channel fusion strategies, largely neglecting the intrinsic material information in hyperspectral imagery. Moreover, the few material-aware approaches typically rely on external spectral unmixing pipelines that are decoupled from the tracking objective, limiting effective optimization of material representations for target localization. To address these limitations, we formulate hyperspectral object tracking as a joint optimization problem of material decomposition and target localization, coupling the two tasks via a weighted target-oriented unmixing loss that explicitly aligns material representations with localization accuracy. Specifically, we propose a material representation decomposition module for deep learning-based spectral unmixing with adaptive frequency decomposition. Building on the decomposed material representations, we further introduce a dual-branch wavelet-enhanced material prompt module that learns low- and high-frequency material prompts through efficient spatial-material interactions in the frequency domain. The framework is model-agnostic and can be seamlessly generalized to different unmixing backbones. Extensive experiments on standard hyperspectral tracking benchmarks demonstrate state-of-the-art performance and validate the effectiveness of the proposed end-to-end material-aware tracking framework. Code is available at https://github.com/han030927/E2EMPT.

Problem

• hyperspectral object tracking
• spectral unmixing
• material representation
• target localization
• end-to-end learning

Innovation

• end-to-end unmixing
• material prompts
• hyperspectral object tracking
• frequency-domain interaction
• joint optimization