Multi-Granularity Language-Guided Multi-Object Tracking

2024-06-07

arXiv.org

AI Summary

To address the insufficient robustness of vision-only multi-object tracking (MOT) under environmental disturbances—such as occlusion, motion blur, and cross-domain shifts—this paper proposes LG-MOT, a language-guided MOT framework that pioneers the integration of multi-granularity linguistic descriptions (scene-level and instance-level) into visual feature learning. Methodologically, we design a multimodal feature alignment mechanism and a dual-encoder text representation (CLIP + BERT) to construct a unified language–vision joint embedding space, enhanced by contrastive learning. We further introduce a practical paradigm wherein language supervision is required only during training but not at inference. Additionally, we present the first MOT benchmark with both scene- and instance-level linguistic annotations. Experiments demonstrate that LG-MOT achieves a 2.2% IDF1 improvement on DanceTrack, establishes new state-of-the-art performance on MOT17 and SportsMOT, and significantly enhances cross-domain generalization capability.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Most existing multi-object tracking methods typically learn visual tracking features via maximizing dis-similarities of different instances and minimizing similarities of the same instance. While such a feature learning scheme achieves promising performance, learning discriminative features solely based on visual information is challenging especially in case of environmental interference such as occlusion, blur and domain variance. In this work, we argue that multi-modal language-driven features provide complementary information to classical visual features, thereby aiding in improving the robustness to such environmental interference. To this end, we propose a new multi-object tracking framework, named LG-MOT, that explicitly leverages language information at different levels of granularity (scene-and instance-level) and combines it with standard visual features to obtain discriminative representations. To develop LG-MOT, we annotate existing MOT datasets with scene-and instance-level language descriptions. We then encode both instance-and scene-level language information into high-dimensional embeddings, which are utilized to guide the visual features during training. At inference, our LG-MOT uses the standard visual features without relying on annotated language descriptions. Extensive experiments on three benchmarks, MOT17, DanceTrack and SportsMOT, reveal the merits of the proposed contributions leading to state-of-the-art performance. On the DanceTrack test set, our LG-MOT achieves an absolute gain of 2.2% in terms of target object association (IDF1 score), compared to the baseline using only visual features. Further, our LG-MOT exhibits strong cross-domain generalizability. The dataset and code will be available at ~url{https://github.com/WesLee88524/LG-MOT}.

Problem

• Enhancing tracking robustness using multi-modal language features
• Combining scene-and instance-level language with visual data
• Improving object association accuracy in challenging environments

Innovation

• Combines language and visual features for tracking
• Uses multi-granularity language descriptions (scene-and instance-level)
• Enhances robustness to occlusion, blur, and domain variance