S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking

2024-06-04

AI Summary

To address the insufficient robustness of end-to-end 3D multi-object tracking (MOT) in complex scenarios—such as severe occlusion and small objects—this paper proposes S2-Track, the first fully end-to-end learnable 3D MOT framework. Its core contributions are: (1) 2D-prompted query initialization, which fuses 2D detection and depth estimation to guide 3D query generation; (2) an uncertainty-aware probabilistic decoder that jointly models localization accuracy and confidence estimation; and (3) a hierarchical query denoising strategy that enhances training stability and convergence via explicit noise modeling. Evaluated on the nuScenes test set, S2-Track achieves an AMOTA of 66.3%, outperforming the previous best end-to-end method by 8.9 percentage points and ranking first on the official leaderboard.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: State Estimation
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Tracking, profiling, and countermeasures against them
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

3D multiple object tracking (MOT) plays a crucial role in autonomous driving perception. Recent end-to-end query-based trackers simultaneously detect and track objects, which have shown promising potential for the 3D MOT task. However, existing methods are still in the early stages of development and lack systematic improvements, failing to track objects in certain complex scenarios, like occlusions and the small size of target object's situations. In this paper, we first summarize the current end-to-end 3D MOT framework by decomposing it into three constituent parts: query initialization, query propagation, and query matching. Then we propose corresponding improvements, which lead to a strong yet simple tracker: S2-Track. Specifically, for query initialization, we present 2D-Prompted Query Initialization, which leverages predicted 2D object and depth information to prompt an initial estimate of the object's 3D location. For query propagation, we introduce an Uncertainty-aware Probabilistic Decoder to capture the uncertainty of complex environment in object prediction with probabilistic attention. For query matching, we propose a Hierarchical Query Denoising strategy to enhance training robustness and convergence. As a result, our S2-Track achieves state-of-the-art performance on nuScenes benchmark, i.e., 66.3% AMOTA on test split, surpassing the previous best end-to-end solution by a significant margin of 8.9% AMOTA. We achieve 1st place on the nuScenes tracking task leaderboard.

Problem

• Improving 3D MOT in complex scenarios like occlusions
• Enhancing query initialization with 2D and depth data
• Boosting tracking robustness via hierarchical query denoising

Innovation

• 2D-Prompted Query Initialization for 3D location
• Uncertainty-aware Probabilistic Decoder for prediction
• Hierarchical Query Denoising for training robustness