Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better

2025-03-25

AI Summary

In video prediction, large motions and long-range temporal dependencies often cause feature misalignment, temporal inconsistency, and visual artifacts. To address this, we propose Tracktention—a novel spatiotemporal attention layer that explicitly embeds sparse point trajectory estimation into the attention mechanism, enabling fine-grained inter-frame feature alignment through motion-aware modeling. Lightweight and modular, Tracktention is plug-and-play: it requires no architectural modifications to existing image-based models, yet elevates them to high-performance video predictors. Our approach integrates trajectory-guided attention with a Vision Transformer backbone and is validated on video depth estimation and colorization tasks. Results demonstrate substantial improvements in temporal consistency, outperforming native video models while incurring minimal computational overhead and exhibiting strong generalization across diverse video understanding tasks.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Temporal consistency is critical in video prediction to ensure that outputs are coherent and free of artifacts. Traditional methods, such as temporal attention and 3D convolution, may struggle with significant object motion and may not capture long-range temporal dependencies in dynamic scenes. To address this gap, we propose the Tracktention Layer, a novel architectural component that explicitly integrates motion information using point tracks, i.e., sequences of corresponding points across frames. By incorporating these motion cues, the Tracktention Layer enhances temporal alignment and effectively handles complex object motions, maintaining consistent feature representations over time. Our approach is computationally efficient and can be seamlessly integrated into existing models, such as Vision Transformers, with minimal modification. It can be used to upgrade image-only models to state-of-the-art video ones, sometimes outperforming models natively designed for video prediction. We demonstrate this on video depth prediction and video colorization, where models augmented with the Tracktention Layer exhibit significantly improved temporal consistency compared to baselines.

Problem

• Enhancing temporal consistency in video prediction
• Addressing object motion challenges in dynamic scenes
• Improving long-range temporal dependency capture

Innovation

• Tracktention Layer integrates motion via point tracks
• Enhances temporal alignment for complex motions
• Efficiently upgrades image models to video models