A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospects

2025-06-16

AI Summary

Video scene parsing (VSP) faces core challenges including maintaining temporal consistency, modeling dynamic scenes, and ensuring robustness in real-world deployment. This paper presents the first systematic survey of state-of-the-art VSP research, covering video semantic/instance/panoptic segmentation, joint tracking-and-segmentation, and open-vocabulary segmentation. It traces the paradigm shift from handcrafted features to CNNs and Transformers. Methodologically, it integrates deep learning with temporal modeling, multi-task learning, and open-vocabulary representation to identify persistent technical bottlenecks. Key contributions include: (1) a unified formalization of VSP task boundaries; (2) a cross-task technological evolution map; and (3) a challenge taxonomy grounded in realistic scenarios, coupled with a critical benchmarking evaluation framework. Finally, it identifies six emerging directions—lightweight temporal modeling, embodied VSP, generative scene parsing, among others—providing a comprehensive roadmap for both academic research and industrial deployment.

Technology Category

• Computer Vision: Scene Analysis & Understanding
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Video Scene Parsing (VSP) has emerged as a cornerstone in computer vision, facilitating the simultaneous segmentation, recognition, and tracking of diverse visual entities in dynamic scenes. In this survey, we present a holistic review of recent advances in VSP, covering a wide array of vision tasks, including Video Semantic Segmentation (VSS), Video Instance Segmentation (VIS), Video Panoptic Segmentation (VPS), as well as Video Tracking and Segmentation (VTS), and Open-Vocabulary Video Segmentation (OVVS). We systematically analyze the evolution from traditional hand-crafted features to modern deep learning paradigms -- spanning from fully convolutional networks to the latest transformer-based architectures -- and assess their effectiveness in capturing both local and global temporal contexts. Furthermore, our review critically discusses the technical challenges, ranging from maintaining temporal consistency to handling complex scene dynamics, and offers a comprehensive comparative study of datasets and evaluation metrics that have shaped current benchmarking standards. By distilling the key contributions and shortcomings of state-of-the-art methodologies, this survey highlights emerging trends and prospective research directions that promise to further elevate the robustness and adaptability of VSP in real-world applications.

Problem

• Review advances in video scene parsing tasks and techniques
• Analyze challenges like temporal consistency and scene dynamics
• Identify future research directions for robust video parsing

Innovation

• Deep learning paradigms for video scene parsing
• Transformer-based architectures for temporal contexts
• Comprehensive datasets and evaluation metrics