Exploiting Temporal State Space Sharing for Video Semantic Segmentation

2025-03-26

AI Summary

To address the inefficiency in temporal modeling, high memory overhead, and poor adaptability to long sequences in video semantic segmentation (VSS), this paper proposes TV3S—a novel architecture built upon the Mamba state-space model. TV3S introduces a selective state gating mechanism to enable lightweight temporal information propagation, incorporates shifted spatial patch operations to support parallel computation and implicit long-range temporal modeling without explicit feature pooling, and establishes an end-to-end cross-frame dynamic feature sharing pathway. Evaluated on VSPW and Cityscapes, TV3S significantly outperforms existing methods, achieving both high segmentation accuracy and improved inference efficiency. It effectively balances accuracy and real-time performance for long-video segmentation, offering a new paradigm for efficient video understanding.

Technology Category

• Computer Vision: Segmentation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Video semantic segmentation (VSS) plays a vital role in understanding the temporal evolution of scenes. Traditional methods often segment videos frame-by-frame or in a short temporal window, leading to limited temporal context, redundant computations, and heavy memory requirements. To this end, we introduce a Temporal Video State Space Sharing (TV3S) architecture to leverage Mamba state space models for temporal feature sharing. Our model features a selective gating mechanism that efficiently propagates relevant information across video frames, eliminating the need for a memory-heavy feature pool. By processing spatial patches independently and incorporating shifted operation, TV3S supports highly parallel computation in both training and inference stages, which reduces the delay in sequential state space processing and improves the scalability for long video sequences. Moreover, TV3S incorporates information from prior frames during inference, achieving long-range temporal coherence and superior adaptability to extended sequences. Evaluations on the VSPW and Cityscapes datasets reveal that our approach outperforms current state-of-the-art methods, establishing a new standard for VSS with consistent results across long video sequences. By achieving a good balance between accuracy and efficiency, TV3S shows a significant advancement in spatiotemporal modeling, paving the way for efficient video analysis. The code is publicly available at https://github.com/Ashesham/TV3S.git.

Problem

• Enhance temporal context in video semantic segmentation
• Reduce redundant computations and memory requirements
• Improve scalability for long video sequences

Innovation

• Temporal state space sharing for VSS
• Selective gating mechanism for efficiency
• Parallel computation with shifted operation