Segmenting, Fast and Slow: Real-Time Open-Vocabulary Video Instance Segmentation with Dual-Path Processing

2026-06-30

AI Summary

This work addresses the challenge of achieving both real-time performance and high accuracy in open-vocabulary video instance segmentation on mobile devices. To this end, the authors propose SegFS, a dual-path framework that leverages an open-vocabulary object model on sparse keyframes to generate instance representations, which are then projected into feature space to condition a lightweight fast branch network. This design enables efficient temporal propagation and dense mask prediction. By shifting instance propagation from object decoding to feature space and decoupling multimodal semantic understanding from the segmentation task, SegFS substantially improves computational efficiency. Experiments demonstrate that the fast branch of SegFS reduces latency by up to 14× compared to the mobile-optimized MOBIUS baseline while maintaining competitive performance on standard open-vocabulary video instance segmentation benchmarks.

Abstract

Object-centric models inspired by DETR have become the dominant paradigm for open-vocabulary video instance segmentation (OV-VIS). While recent efforts have reduced the computational cost of pixel decoding, textual modality fusion, and object decoding to make these architectures more suitable for mobile devices, real-time on-device inference at high frame rates remains an open challenge. In this paper, we introduce SegFS, a dual-stream fast-slow framework that significantly improves efficiency without sacrificing accuracy. On sparse keyframes, an open-vocabulary object-based model predicts instance-level representations. These representations are then projected back into the backbone feature space to condition a lightweight fast network, which efficiently relocalizes and segments the instances in subsequent frames. By shifting instance propagation from object decoding to feature-space conditioning, our approach decouples multimodal semantic understanding from dense mask prediction and enables efficient temporal propagation. The proposed fast branch achieves up to 14x lower latency than the mobile-oriented MOBIUS model, while maintaining competitive segmentation performance on standard OV-VIS benchmarks.

Problem

• open-vocabulary video instance segmentation
• real-time inference
• on-device processing
• video instance segmentation
• computational efficiency

Innovation

• dual-path processing
• open-vocabulary video instance segmentation
• feature-space conditioning
• real-time inference
• instance propagation