VidEoMT: Your ViT is Secretly Also a Video Segmentation Model

2026-02-19

AI Summary

This work proposes VidEoMT, a novel approach to video object segmentation that eliminates the need for complex, dedicated tracking modules commonly used in existing methods, thereby reducing architectural redundancy and computational overhead. For the first time, efficient video segmentation is achieved within a pure Vision Transformer (ViT) framework by introducing a lightweight query propagation and fusion mechanism to model temporal consistency across frames. Operating on a ViT-L backbone, the method attains a real-time inference speed of 160 FPS—5 to 10 times faster than current state-of-the-art approaches—while maintaining competitive segmentation accuracy. This advancement significantly lowers model complexity without sacrificing performance, demonstrating the efficacy of integrating temporal modeling directly into the transformer architecture.

Technology Category

• Computer Vision: Segmentation
• Search and Optimization: Learning to Search
• Machine Learning: Mixture of Experts (MoE)

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x--10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/

Problem

• video segmentation
• tracking modules
• architectural complexity
• computational overhead

Innovation

• Vision Transformer
• video segmentation
• query propagation
• encoder-only architecture
• temporal modeling