Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation

2025-06-11

AI Summary

This work addresses the joint challenge of open-vocabulary semantic segmentation (OVSS) and domain-generalized semantic segmentation (DGSS), proposing Vireo—the first unified framework for robust pixel-level segmentation of both unseen categories and unseen domains. Methodologically: (1) GeoText prompt alignment jointly models geometric structure and linguistic semantics; (2) coarse mask prior embedding is introduced to strengthen gradient propagation; (3) a domain–open-vocabulary vector embedding head enables cross-domain feature disentanglement and multimodal semantic fusion. Leveraging a frozen vision foundation model and deep geometric modeling, Vireo achieves state-of-the-art performance on both OVSS and DGSS benchmarks, significantly outperforming existing approaches. The code is publicly available.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Open-Vocabulary semantic segmentation (OVSS) and domain generalization in semantic segmentation (DGSS) highlight a subtle complementarity that motivates Open-Vocabulary Domain-Generalized Semantic Segmentation (OV-DGSS). OV-DGSS aims to generate pixel-level masks for unseen categories while maintaining robustness across unseen domains, a critical capability for real-world scenarios such as autonomous driving in adverse conditions. We introduce Vireo, a novel single-stage framework for OV-DGSS that unifies the strengths of OVSS and DGSS for the first time. Vireo builds upon the frozen Visual Foundation Models (VFMs) and incorporates scene geometry via Depth VFMs to extract domain-invariant structural features. To bridge the gap between visual and textual modalities under domain shift, we propose three key components: (1) GeoText Prompts, which align geometric features with language cues and progressively refine VFM encoder representations; (2) Coarse Mask Prior Embedding (CMPE) for enhancing gradient flow for faster convergence and stronger textual influence; and (3) the Domain-Open-Vocabulary Vector Embedding Head (DOV-VEH), which fuses refined structural and semantic features for robust prediction. Comprehensive evaluation on these components demonstrates the effectiveness of our designs. Our proposed Vireo achieves the state-of-the-art performance and surpasses existing methods by a large margin in both domain generalization and open-vocabulary recognition, offering a unified and scalable solution for robust visual understanding in diverse and dynamic environments. Code is available at https://github.com/anonymouse-9c53tp182bvz/Vireo.

Problem

• Enhance semantic segmentation for unseen categories and domains
• Unify open-vocabulary and domain-generalized segmentation capabilities
• Improve robustness in dynamic real-world scenarios like autonomous driving

Innovation

• Leverages Depth VFMs for domain-invariant features
• Introduces GeoText Prompts for modality alignment
• Uses DOV-VEH for robust semantic-structural fusion