SAM-MT: Real-Time Interactive Multi-Target Video Segmentation

2026-07-09

AI Summary

Existing interactive multi-object video segmentation methods suffer from significant latency increases with the number of objects due to processing each target independently, hindering real-time performance. To address this, this work proposes an efficient parallel framework built upon Segment Anything Model 2 (SAM2), which jointly models multiple objects through explicit object queries and a shared global context. The approach introduces a decoupled mask attention mechanism and a sparse memory structure to effectively mitigate interference and occlusion among targets. Combined with an overlap suppression strategy, the method maintains SAM2-level segmentation accuracy while decoupling inference latency from the number of objects, achieving over 36 FPS even with ten simultaneous targets and enabling real-time multi-object interaction.

Abstract

Modern Video Object Segmentation (VOS) involves tracking and segmenting user-specified targets. While recent approaches have achieved remarkable performance in single-target scenarios, extending them to multi-target settings typically involves replicating the single-target processing for each individual object, resulting in reduced frame rates (FPS) with unbounded latency as target count increases. Built upon Segment Anything 2 (SAM2), we propose SAM-MT, which addresses this by transforming the model into an interactive framework for real-time Multi-Target video segmentation. SAM-MT uses explicit queries to represent different individual targets, in parallel with a shared representation for global context. It employs decoupled masked attention to keep individual identities distinct from cross-target interference, and sparse memory for stable temporal evolution, along with specialized strategies for occlusion handling and overlap prevention. SAM-MT successfully decouples latency from the number of targets, achieving real-time speed on par with single-target baselines (>36 FPS for 10 targets) while maintaining SAM2's robust video segmentation performance.

Problem

• Video Object Segmentation
• Multi-Target
• Real-Time
• Latency
• Interactive

Innovation

• multi-target video segmentation
• real-time interaction
• decoupled masked attention
• sparse memory
• occlusion handling