Autoregressive Universal Video Segmentation Model

2025-08-26

AI Summary

Existing video segmentation methods perform well in prompted settings but lack a unified framework for prompt-free (fully automatic) detection and tracking. This work proposes the Autoregressive Universal Segmentation Model (AUSM), the first to introduce State Space Models (SSMs) into video segmentation, reformulating it as a sequential mask prediction task to jointly handle both prompted and prompt-free scenarios. AUSM employs a fixed-dimensional state representation, enabling efficient processing of arbitrarily long video sequences while supporting cross-frame parallel training and sequential inference. Evaluated on standard benchmarks—including DAVIS17 and YouTube-VOS—AUSM achieves state-of-the-art performance. Moreover, it attains up to 2.5× faster training on 16-frame sequences compared to prior methods, significantly enhancing both model generality and computational efficiency.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Recent video foundation models such as SAM2 excel at prompted video segmentation by treating masks as a general-purpose primitive. However, many real-world settings require unprompted segmentation that aims to detect and track all objects in a video without external cues, leaving today's landscape fragmented across task-specific models and pipelines. We recast streaming video segmentation as sequential mask prediction, analogous to language modeling, and introduce the Autoregressive Universal Segmentation Model (AUSM), a single architecture that unifies both prompted and unprompted video segmentation. Built on recent state-space models, AUSM maintains a fixed-size spatial state and scales to video streams of arbitrary length. Furthermore, all components of AUSM are designed for parallel training across frames, yielding substantial speedups over iterative training. On standard benchmarks (DAVIS17, YouTube-VOS 2018 & 2019, MOSE, YouTube-VIS 2019 & 2021, and OVIS) AUSM outperforms prior universal streaming video segmentation methods and achieves up to 2.5x faster training on 16-frame sequences.

Problem

• Unifying prompted and unprompted video segmentation tasks
• Detecting and tracking all objects without external cues
• Addressing fragmented task-specific models with single architecture

Innovation

• Autoregressive sequential mask prediction model
• Unified prompted and unprompted segmentation architecture
• Parallel training design for faster processing