TAPNext++: What's Next for Tracking Any Point (TAP)?

2026-04-12

AI Summary

This work addresses the limitations of existing TAP models, which suffer from degraded tracking performance on long videos and struggle to re-detect points that reappear after occlusion or exiting the frame. To overcome these challenges, we propose TAPNext++, a recursive video Transformer architecture that integrates sequence-parallel training—supporting sequences up to 1,024 frames—and data-driven geometric augmentations, such as periodic rolling, to simulate point re-entry. Additionally, an occlusion-aware supervision mechanism is introduced to enhance re-detection robustness. We also present a new evaluation metric, Re-Detection Average Jaccard (AJ_RD), to specifically assess re-detection performance. TAPNext++ achieves state-of-the-art results across multiple benchmarks, significantly outperforming prior methods in both long-sequence tracking and point re-detection while maintaining low memory and computational overhead.

Technology Category

• Computer Vision: Motion & Tracking
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Tracking-Any-Point (TAP) models aim to track any point through a video which is a crucial task in AR/XR and robotics applications. The recently introduced TAPNext approach proposes an end-to-end, recurrent transformer architecture to track points frame-by-frame in a purely online fashion -- demonstrating competitive performance at minimal latency. However, we show that TAPNext struggles with longer video sequences and also frequently fails to re-detect query points that reappear after being occluded or leaving the frame. In this work, we present TAPNext++, a model that tracks points in sequences that are orders of magnitude longer while preserving the low memory and compute footprint of the architecture. We train the recurrent video transformer using several data-driven solutions, including training on long 1024-frame sequences enabled by sequence parallelism techniques. We highlight that re-detection performance is a blind spot in the current literature and introduce a new metric, Re-Detection Average Jaccard ($AJ_{RD}$), to explicitly evaluate tracking on re-appearing points. To improve re-detection of points, we introduce tailored geometric augmentations, such as periodic roll that simulates point re-entries, and supervising occluded points. We demonstrate that recurrent transformers can be substantially improved for point tracking and set a new state-of-the-art on multiple benchmarks. Model and code can be found at https://tap-next-plus-plus.github.io.

Problem

• point tracking
• occlusion handling
• long video sequences
• re-detection
• video transformers

Innovation

• point tracking
• recurrent transformer
• re-detection
• sequence parallelism
• geometric augmentation