DELTAv2: Accelerating Dense 3D Tracking

2025-08-01

AI Summary

To address the high computational cost and poor real-time performance of dense, long-term 3D point tracking in videos, this paper proposes an efficient tracking framework. Methodologically, it introduces a coarse-to-fine iterative tracking strategy, an end-to-end learnable Transformer-based interpolation module, and optimized feature extraction and matching procedures; additionally, a progressive trajectory expansion mechanism is incorporated to significantly reduce redundant computations. The core contribution lies in formulating interpolation as a learnable spatiotemporal relational reasoning task, jointly optimizing feature representation and the tracking pipeline. Extensive experiments demonstrate that the framework achieves state-of-the-art (SOTA) accuracy while accelerating inference by 5–100× over prior methods—enabling, for the first time, millisecond-level dense long-term 3D point tracking. This breakthrough establishes a new paradigm for real-time applications such as AR/VR and robotic vision.

Technology Category

• Computer Vision: Motion & Tracking
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Tracking, profiling, and countermeasures against them
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We propose a novel algorithm for accelerating dense long-term 3D point tracking in videos. Through analysis of existing state-of-the-art methods, we identify two major computational bottlenecks. First, transformer-based iterative tracking becomes expensive when handling a large number of trajectories. To address this, we introduce a coarse-to-fine strategy that begins tracking with a small subset of points and progressively expands the set of tracked trajectories. The newly added trajectories are initialized using a learnable interpolation module, which is trained end-to-end alongside the tracking network. Second, we propose an optimization that significantly reduces the cost of correlation feature computation, another key bottleneck in prior methods. Together, these improvements lead to a 5-100x speedup over existing approaches while maintaining state-of-the-art tracking accuracy.

Problem

• Accelerating dense long-term 3D point tracking
• Reducing computational cost of transformer-based tracking
• Optimizing correlation feature computation efficiency

Innovation

• Coarse-to-fine strategy for trajectory tracking
• Learnable interpolation module for initialization
• Optimized correlation feature computation method