TAPNext: Tracking Any Point (TAP) as Next Token Prediction

2025-04-08

AI Summary

Existing methods for Track Any Point (TAP) in videos suffer from limited generalization and real-time performance due to reliance on strong inductive biases and hand-crafted heuristics. To address this, we reformulate TAP as a causal masked token sequence prediction task—its first such formulation. We propose a purely online, streaming Transformer architecture that eliminates temporal window constraints and tracking-specific priors, enabling trajectory generation to emerge naturally through end-to-end training. Our core innovation lies in redefining point tracking as autoregressive frame-by-frame prediction of discrete masked tokens corresponding to point locations, yielding low-latency and highly generalizable trajectory estimation. Evaluated on the TAP benchmark, our method achieves state-of-the-art performance, outperforming both existing online and offline trackers while significantly reducing inference latency.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Time-Series/Data Streams
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Tracking, profiling, and countermeasures against them
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Tracking Any Point (TAP) in a video is a challenging computer vision problem with many demonstrated applications in robotics, video editing, and 3D reconstruction. Existing methods for TAP rely heavily on complex tracking-specific inductive biases and heuristics, limiting their generality and potential for scaling. To address these challenges, we present TAPNext, a new approach that casts TAP as sequential masked token decoding. Our model is causal, tracks in a purely online fashion, and removes tracking-specific inductive biases. This enables TAPNext to run with minimal latency, and removes the temporal windowing required by many existing state of art trackers. Despite its simplicity, TAPNext achieves a new state-of-the-art tracking performance among both online and offline trackers. Finally, we present evidence that many widely used tracking heuristics emerge naturally in TAPNext through end-to-end training.

Problem

• TAPNext addresses video point tracking challenges
• It removes complex tracking-specific biases
• Achieves state-of-the-art online and offline performance

Innovation

• Sequential masked token decoding for TAP
• Causal online tracking without biases
• End-to-end training replaces tracking heuristics