FeedbackTrack: Visual-Cortex-Inspired Cross-Frame Feedback for Transformer Tracking

2026-08-10

AI Summary

This work addresses the challenge that existing Transformer-based trackers struggle to effectively integrate temporal information due to the absence of cross-frame reuse of intermediate features. Inspired by the visual cortex, we introduce, for the first time in Transformer tracking, a sparse and group-aligned cross-frame feedback mechanism. This mechanism employs two lightweight pathways—Query Feedback and Gate Feedback—to cache intermediate states from the previous frame and reinject them into corresponding modules of the current frame. Our approach reveals that feedback strength exhibits depth-dependent characteristics and demonstrates that recycling historical information consistently outperforms single-frame modulation. Evaluated on SPMTrack and ARTrackV2, the proposed method uniformly enhances performance across five backbone architectures, achieving 83.4 AO on LaSOT and 79.1 AUC on GOT-10k, with less than 1% parameter overhead and gains of 1.8–3.2 AO points over same-frame modulation baselines.

Abstract

Visual object tracking requires effective temporal integration, yet most Transformer trackers still rely on predominantly feed-forward feature extraction. Existing temporal mechanisms typically update templates, prompts, queries, or prediction states, while intermediate representations are rarely reused to modulate corresponding processing stages. We propose \textbf{FeedbackTrack}, a visual-cortex-inspired framework that introduces sparse, group-level layer-aligned cross-frame feedback into pretrained Transformer trackers. Previous-frame intermediate states are detached, cached, and returned to corresponding Transformer groups in the current frame through two lightweight pathways: Query Feedback for token-level query modulation and Gate Feedback for context-dependent feature modulation. FeedbackTrack preserves the original tracking pipeline with only a fixed-size one-frame cache. Across SPMTrack and ARTrackV2, FeedbackTrack consistently improves five backbone configurations on LaSOT and GOT-10k, achieving 83.4 AO and 79.1 AUC with SPMTrack-G while adding less than 1\% parameters. Controlled comparisons show that cross-frame feedback outperforms same-frame modulation by 1.8--3.2 AO points, demonstrating that the gains mainly come from recurrent historical information. Further analysis reveals a non-uniform depth-dependent organization of learned feedback strengths, highlighting the effectiveness of recurrent feedback for Transformer tracking.

Problem

• visual object tracking
• Transformer tracking
• temporal integration
• cross-frame feedback
• intermediate representations

Innovation

• cross-frame feedback
• Transformer tracking
• visual cortex inspiration
• recurrent temporal integration
• intermediate feature reuse