SeC: Advancing Complex Video Object Segmentation via Progressive Concept Construction

2025-07-21

AI Summary

Current video object segmentation (VOS) methods rely heavily on low-level appearance matching, rendering them vulnerable to severe deformations, occlusions, and scene dynamics, while lacking human-like high-level semantic understanding. To address this, we propose ConceptSeg—a novel framework that replaces appearance-based matching with semantic concept representations. ConceptSeg introduces three key components: LVLM-driven cross-frame concept construction, concept-enhanced feature matching, and adaptive higher-order semantic reasoning. To advance research in conceptual VOS, we introduce SeCVOS—the first benchmark explicitly designed for semantically complex video scenes. On SeCVOS, ConceptSeg achieves a 11.8 mAP gain over SAM 2.1, significantly improving robustness and stability in dynamic scenarios. This work establishes the first concept-aware VOS paradigm, marking a fundamental shift from pixel- or feature-level matching toward semantics-guided video segmentation.

Technology Category

• Computer Vision: Segmentation
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Video Object Segmentation (VOS) is a core task in computer vision, requiring models to track and segment target objects across video frames. Despite notable advances with recent efforts, current techniques still lag behind human capabilities in handling drastic visual variations, occlusions, and complex scene changes. This limitation arises from their reliance on appearance matching, neglecting the human-like conceptual understanding of objects that enables robust identification across temporal dynamics. Motivated by this gap, we propose Segment Concept (SeC), a concept-driven segmentation framework that shifts from conventional feature matching to the progressive construction and utilization of high-level, object-centric representations. SeC employs Large Vision-Language Models (LVLMs) to integrate visual cues across diverse frames, constructing robust conceptual priors. During inference, SeC forms a comprehensive semantic representation of the target based on processed frames, realizing robust segmentation of follow-up frames. Furthermore, SeC adaptively balances LVLM-based semantic reasoning with enhanced feature matching, dynamically adjusting computational efforts based on scene complexity. To rigorously assess VOS methods in scenarios demanding high-level conceptual reasoning and robust semantic understanding, we introduce the Semantic Complex Scenarios Video Object Segmentation benchmark (SeCVOS). SeCVOS comprises 160 manually annotated multi-scenario videos designed to challenge models with substantial appearance variations and dynamic scene transformations. In particular, SeC achieves an 11.8-point improvement over SAM 2.1 on SeCVOS, establishing a new state-of-the-art in concept-aware video object segmentation.

Problem

• Handling drastic visual variations and occlusions in VOS
• Shifting from feature matching to concept-driven segmentation
• Improving semantic reasoning for complex scene changes

Innovation

• Progressive construction of object-centric representations
• Integration of LVLMs for robust conceptual priors
• Dynamic balance of semantic reasoning and feature matching