Online 3D Multi-Camera Perception through Robust 2D Tracking and Depth-based Late Aggregation

2025-09-11

AI Summary

To address the challenge of extending 2D multi-target multi-camera tracking (MTMC) systems to 3D space, this paper proposes a lightweight, plug-and-play 3D extension framework. It leverages monocular depth estimation and camera calibration to project online 2D detections and trajectories into point-cloud space. Cross-camera data association is enhanced via local identity consistency constraints, while a depth-aware post-fusion strategy ensures global identity coherence. Furthermore, point-cloud clustering and yaw-angle optimization are integrated to reconstruct high-accuracy 3D bounding boxes and trajectories. Crucially, the method operates without modifying the underlying 2D tracking modules, significantly reducing deployment overhead. Evaluated on the 2025 AI City Challenge 3D MTMC benchmark, it achieves third place—demonstrating strong effectiveness, robustness, and engineering practicality.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Multi-Target Multi-Camera Tracking (MTMC) is an essential computer vision task for automating large-scale surveillance. With camera calibration and depth information, the targets in the scene can be projected into 3D space, offering unparalleled levels of automatic perception of a 3D environment. However, tracking in the 3D space requires replacing all 2D tracking components from the ground up, which may be infeasible for existing MTMC systems. In this paper, we present an approach for extending any online 2D multi-camera tracking system into 3D space by utilizing depth information to reconstruct a target in point-cloud space, and recovering its 3D box through clustering and yaw refinement following tracking. We also introduced an enhanced online data association mechanism that leverages the target's local ID consistency to assign global IDs across frames. The proposed framework is evaluated on the 2025 AI City Challenge's 3D MTMC dataset, achieving 3rd place on the leaderboard.

Problem

• Extend 2D multi-camera tracking to 3D space
• Reconstruct target point-clouds using depth information
• Assign global IDs across frames with enhanced association

Innovation

• Depth-based late aggregation for 3D reconstruction
• Clustering and yaw refinement for 3D boxes
• Enhanced online data association mechanism