Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

2026-08-04

AI Summary

This work addresses the performance degradation in multi-camera 3D object tracking using only RGB images under Sim2Real domain shifts, primarily caused by the absence of reliable depth cues. The authors propose and validate a geometry-consistency-first strategy that outperforms pseudo-LiDAR approaches based on monocular depth estimation. They demonstrate for the first time that the cross-view inconsistency inherent in monocular depth estimation constitutes a key bottleneck in Sim2Real transfer. Their geometrically grounded pipeline integrates YOLO11x detection, homography lifting, category-level size priors, multi-view fusion, and world-coordinate tracking. Experimental results show that this approach achieves a 3D HOTA score of 13.0 (LocA: 51.6), substantially surpassing the pseudo-LiDAR baseline at 0.12 HOTA (LocA: 9.2), thereby highlighting the critical role of geometric consistency in ensuring Sim2Real robustness.

Abstract

The AI City Challenge 2026 Track 1 evaluates multi-camera 3D perception in large indoor warehouses under a synthetic-to-real (Sim2Real) setting; depth is available only for training and validation, so inference is RGB-only. We use two RGB-only routes as a controlled test of one hypothesis: that cross-view geometric consistency, not monocular depth accuracy, governs performance under Sim2Real. The first is a geometry-first pipeline: YOLO11x detection, homography lifting to the world frame, class-level 3D size priors, multi-camera fusion, world-coordinate tracking, and offline tracklet stitching. The second is estimated-depth pseudo-LiDAR: monocular depth (D4RT, Metric3D~v2) back-projected into a fused point cloud and passed to a 3D detector (V-DETR), mirroring prior point-cloud winners that used depth. The gap is decisive: geometry-first reaches 13.0 3D HOTA (51.6 LocA), whereas pseudo-LiDAR collapses to 0.12 (9.2 LocA). We trace the collapse to cross-view inconsistency of monocular depth---scale correction is necessary but not sufficient---which domain-adaptation fine-tuning does not repair within budget. Within the geometry pipeline, offline stitching is the only intervention that helps; SAHI detection, appearance Re-ID, learned lifting, RT-DETR ensembling, test-time augmentation, and domain randomization all fail to beat the baseline detector. The bottlenecks are complementary: detection quality bounds the geometry route (DetA), localization consistency bounds pseudo-LiDAR (LocA). We release a complete, reproducible RGB-only pipeline and ablation.

Problem

• Sim2Real
• multi-camera 3D tracking
• monocular depth estimation
• cross-view consistency
• RGB-only perception

Innovation

• geometric consistency
• Sim2Real
• RGB-only 3D tracking
• pseudo-LiDAR
• multi-camera fusion