Seg2Track-SAM2: SAM2-based Multi-object Tracking and Segmentation for Zero-shot Generalization

2025-09-15

AI Summary

To address weak identity consistency and high memory overhead in multi-object tracking and segmentation (MOTS) for dynamic scenes, this paper proposes a fine-tuning-free zero-shot framework. It couples a pre-trained detector with SAM2 and introduces a lightweight Seg2Track module to jointly handle trajectory initialization, association, and enhancement—significantly improving ID continuity. A sliding-window memory strategy is further adopted, reducing GPU memory consumption by 75%. The method is detector-agnostic and generalizable to video instance segmentation. Evaluated on the KITTI MOTS benchmark, it ranks fourth overall and achieves state-of-the-art performance on both vehicle and pedestrian classes, with substantial gains in association accuracy (AssA). These results demonstrate its computational efficiency and strong generalization capability across object categories and detection backbones.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Tracking, profiling, and countermeasures against them
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Autonomous systems require robust Multi-Object Tracking (MOT) capabilities to operate reliably in dynamic environments. MOT ensures consistent object identity assignment and precise spatial delineation. Recent advances in foundation models, such as SAM2, have demonstrated strong zero-shot generalization for video segmentation, but their direct application to MOTS (MOT+Segmentation) remains limited by insufficient identity management and memory efficiency. This work introduces Seg2Track-SAM2, a framework that integrates pre-trained object detectors with SAM2 and a novel Seg2Track module to address track initialization, track management, and reinforcement. The proposed approach requires no fine-tuning and remains detector-agnostic. Experimental results on KITTI MOT and KITTI MOTS benchmarks show that Seg2Track-SAM2 achieves state-of-the-art (SOTA) performance, ranking fourth overall in both car and pedestrian classes on KITTI MOTS, while establishing a new benchmark in association accuracy (AssA). Furthermore, a sliding-window memory strategy reduces memory usage by up to 75% with negligible performance degradation, supporting deployment under resource constraints. These results confirm that Seg2Track-SAM2 advances MOTS by combining robust zero-shot tracking, enhanced identity preservation, and efficient memory utilization. The code is available at https://github.com/hcmr-lab/Seg2Track-SAM2

Problem

• Integrating SAM2 for zero-shot multi-object tracking and segmentation
• Addressing identity management and memory efficiency limitations in MOTS
• Achieving robust tracking without fine-tuning or detector dependency

Innovation

• Integrates SAM2 with object detectors
• Uses Seg2Track module for identity management
• Implements sliding-window memory strategy