DELTA: Dense Efficient Long-range 3D Tracking for any video

2024-10-31

arXiv.org

AI Summary

This work addresses the challenging problem of long-term, pixel-level dense 3D motion tracking from monocular video. Methodologically, we propose the first end-to-end framework for full-pixel 3D trajectory estimation, featuring a global-local joint attention mechanism to model long-range spatiotemporal dependencies; a lightweight Transformer-based upsampler for efficient high-resolution reconstruction; and log-depth encoding—demonstrated as the optimal depth representation—integrated within a hybrid paradigm combining low-resolution tracking and high-resolution reconstruction. Our contributions are threefold: (1) the first method achieving high-accuracy, high-efficiency, and truly dense long-term 3D tracking; (2) new state-of-the-art results on multiple benchmarks, with superior 2D and 3D dense tracking accuracy; and (3) inference speed over eight times faster than prior best methods.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Deep Generative Models & Autoencoders
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Security and Privacy: Tracking, profiling, and countermeasures against them

Abstract

Tracking dense 3D motion from monocular videos remains challenging, particularly when aiming for pixel-level precision over long sequences. We introduce DELTA, a novel method that efficiently tracks every pixel in 3D space, enabling accurate motion estimation across entire videos. Our approach leverages a joint global-local attention mechanism for reduced-resolution tracking, followed by a transformer-based upsampler to achieve high-resolution predictions. Unlike existing methods, which are limited by computational inefficiency or sparse tracking, DELTA delivers dense 3D tracking at scale, running over 8x faster than previous methods while achieving state-of-the-art accuracy. Furthermore, we explore the impact of depth representation on tracking performance and identify log-depth as the optimal choice. Extensive experiments demonstrate the superiority of DELTA on multiple benchmarks, achieving new state-of-the-art results in both 2D and 3D dense tracking tasks. Our method provides a robust solution for applications requiring fine-grained, long-term motion tracking in 3D space.

Problem

• Efficient dense 3D motion tracking from monocular videos.
• Achieving pixel-level precision over long video sequences.
• Improving computational efficiency and accuracy in 3D tracking.

Innovation

• Joint global-local attention mechanism for tracking
• Transformer-based upsampler for high-resolution predictions
• Log-depth representation for optimal tracking performance