Fast Self-Supervised depth and mask aware Association for Multi-Object Tracking

2025-10-10

AI Summary

In multi-object tracking (MOT), IoU-based association suffers from poor robustness under severe occlusion or when objects exhibit similar appearance, while mask-IoU computation incurs high overhead due to reliance on segmentation masks. To address this, we propose a lightweight association method that fuses depth and mask features: zero-shot depth estimation and promptable segmentation provide geometric and shape cues; a compact encoder—trained self-supervisedly—learns discriminative joint representations to replace explicit mask-IoU computation; finally, bounding-box IoU and re-identification features are integrated into a multi-cue matching strategy. To our knowledge, this is the first work to introduce self-supervised depth-mask joint encoding for association in the tracking-by-detection (TBD) paradigm. Experiments demonstrate significant improvements over state-of-the-art methods on challenging occlusion-heavy benchmarks (e.g., SportsMOT, DanceTrack), while maintaining competitive accuracy and efficiency on standard benchmarks such as MOT17.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Unsupervised & Self-Supervised Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Multi-object tracking (MOT) methods often rely on Intersection-over-Union (IoU) for association. However, this becomes unreliable when objects are similar or occluded. Also, computing IoU for segmentation masks is computationally expensive. In this work, we use segmentation masks to capture object shapes, but we do not compute segmentation IoU. Instead, we fuse depth and mask features and pass them through a compact encoder trained self-supervised. This encoder produces stable object representations, which we use as an additional similarity cue alongside bounding box IoU and re-identification features for matching. We obtain depth maps from a zero-shot depth estimator and object masks from a promptable visual segmentation model to obtain fine-grained spatial cues. Our MOT method is the first to use the self-supervised encoder to refine segmentation masks without computing masks IoU. MOT can be divided into joint detection-ReID (JDR) and tracking-by-detection (TBD) models. The latter are computationally more efficient. Experiments of our TBD method on challenging benchmarks with non-linear motion, occlusion, and crowded scenes, such as SportsMOT and DanceTrack, show that our method outperforms the TBD state-of-the-art on most metrics, while achieving competitive performance on simpler benchmarks with linear motion, such as MOT17.

Problem

• Improving multi-object tracking reliability under occlusion and similar objects
• Eliminating computationally expensive segmentation mask IoU calculations
• Enhancing object matching using fused depth and mask features

Innovation

• Fuses depth and mask features via self-supervised encoder
• Uses zero-shot depth estimator for spatial cues
• Refines segmentation masks without computing mask IoU