ISTASTrack: Bridging ANN and SNN via ISTA Adapter for RGB-Event Tracking

2025-09-12

AI Summary

This paper addresses key challenges in RGB-event heterogeneous tracking—namely, event stream sparsity and asynchrony, as well as the modality and computational paradigm mismatch between artificial neural networks (ANNs) and spiking neural networks (SNNs). To this end, we propose ISTASTrack, the first Transformer-based hybrid tracker. Its core contributions are: (1) an Iterative Shrinkage-Thresholding Algorithm (ISTA)-inspired adapter grounded in sparse representation theory, enabling bidirectional feature interaction between ANN and SNN branches; (2) a temporal downsampling attention module that aligns heterogeneous temporal features across RGB frames and event streams; and (3) an end-to-end trainable architecture jointly leveraging vision Transformers (for spatial context modeling) and spiking Transformers (for spatiotemporal dynamics modeling). ISTASTrack achieves state-of-the-art performance on FE240Hz, VisEvent, COESOT, and FELT benchmarks, demonstrating both high accuracy and energy efficiency—thereby validating the efficacy of hybrid neural architectures for robust visual tracking.

Technology Category

• Computer Vision: Motion & Tracking
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Security and Privacy: Tracking, profiling, and countermeasures against them
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

RGB-Event tracking has become a promising trend in visual object tracking to leverage the complementary strengths of both RGB images and dynamic spike events for improved performance. However, existing artificial neural networks (ANNs) struggle to fully exploit the sparse and asynchronous nature of event streams. Recent efforts toward hybrid architectures combining ANNs and spiking neural networks (SNNs) have emerged as a promising solution in RGB-Event perception, yet effectively fusing features across heterogeneous paradigms remains a challenge. In this work, we propose ISTASTrack, the first transformer-based extbf{A}NN- extbf{S}NN hybrid extbf{Track}er equipped with extbf{ISTA} adapters for RGB-Event tracking. The two-branch model employs a vision transformer to extract spatial context from RGB inputs and a spiking transformer to capture spatio-temporal dynamics from event streams. To bridge the modality and paradigm gap between ANN and SNN features, we systematically design a model-based ISTA adapter for bidirectional feature interaction between the two branches, derived from sparse representation theory by unfolding the iterative shrinkage thresholding algorithm. Additionally, we incorporate a temporal downsampling attention module within the adapter to align multi-step SNN features with single-step ANN features in the latent space, improving temporal fusion. Experimental results on RGB-Event tracking benchmarks, such as FE240hz, VisEvent, COESOT, and FELT, have demonstrated that ISTASTrack achieves state-of-the-art performance while maintaining high energy efficiency, highlighting the effectiveness and practicality of hybrid ANN-SNN designs for robust visual tracking. The code is publicly available at https://github.com/lsying009/ISTASTrack.git.

Problem

• Bridging ANN and SNN feature fusion for RGB-Event tracking
• Exploiting sparse asynchronous event streams in visual tracking
• Aligning heterogeneous temporal features between neural paradigms

Innovation

• ISTA adapters bridge ANN and SNN features
• Transformer extracts spatial and spatio-temporal dynamics
• Temporal downsamning aligns multi-step SNN features