Exploiting Domain Properties in Language-Driven Domain Generalization for Semantic Segmentation

2025-12-03

AI Summary

Existing single-source domain generalization for semantic segmentation (DGSS) based on vision-language models (VLMs) suffers from vision–text semantic misalignment due to fixed-context prompts. To address this, we propose DPMFormer: (1) a domain-aware prompt learning module that dynamically generates input-domain-specific textual prompts; (2) domain-robust consistency training coupled with contrastive learning to explicitly align cross-domain visual features with textual semantics; and (3) texture perturbation augmentation to enhance robustness against style variations. DPMFormer is the first framework to model multi-domain characteristics under a single-source setting without access to target-domain data. It achieves state-of-the-art performance across multiple DGSS benchmarks, significantly improving generalization to unseen domains and segmentation accuracy.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent domain generalized semantic segmentation (DGSS) studies have achieved notable improvements by distilling semantic knowledge from Vision-Language Models (VLMs). However, they overlook the semantic misalignment between visual and textual contexts, which arises due to the rigidity of a fixed context prompt learned on a single source domain. To this end, we present a novel domain generalization framework for semantic segmentation, namely Domain-aware Prompt-driven Masked Transformer (DPMFormer). Firstly, we introduce domain-aware prompt learning to facilitate semantic alignment between visual and textual cues. To capture various domain-specific properties with a single source dataset, we propose domain-aware contrastive learning along with the texture perturbation that diversifies the observable domains. Lastly, to establish a framework resilient against diverse environmental changes, we have proposed the domain-robust consistency learning which guides the model to minimize discrepancies of prediction from original and the augmented images. Through experiments and analyses, we demonstrate the superiority of the proposed framework, which establishes a new state-of-the-art on various DGSS benchmarks. The code is available at https://github.com/jone1222/DPMFormer.

Problem

• Addresses semantic misalignment in vision-language models for segmentation
• Enhances domain generalization with adaptive prompts and contrastive learning
• Improves robustness against environmental changes through consistency learning

Innovation

• Domain-aware prompt learning for semantic alignment
• Texture perturbation with domain-aware contrastive learning
• Domain-robust consistency learning for prediction stability