In defense of the two-stage framework for open-set domain adaptive semantic segmentation

2026-01-04

arXiv.org

AI Summary

This work addresses the challenge in open-set domain adaptive semantic segmentation where existing single-stage methods struggle to simultaneously handle domain adaptation for known classes and detection of unknown classes, often resulting in negative transfer and underfitting. To overcome this, we propose SATS, a two-stage training framework: the first stage disentangles known and unknown categories, while the second stage introduces unknown-aware domain adaptation and hard unknown-exploration data augmentation to mitigate class imbalance and enhance the discovery of truly unknown objects. Additionally, semantic segmentation model alignment is incorporated to refine feature learning. Evaluated on the GTA5→Cityscapes and SYNTHIA→Cityscapes benchmarks, our method achieves H-Score improvements of 3.85% and 18.64%, respectively, substantially outperforming current state-of-the-art approaches.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Segmentation
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Open-Set Domain Adaptation for Semantic Segmentation (OSDA-SS) presents a significant challenge, as it requires both domain adaptation for known classes and the distinction of unknowns. Existing methods attempt to address both tasks within a single unified stage. We question this design, as the annotation imbalance between known and unknown classes often leads to negative transfer of known classes and underfitting for unknowns. To overcome these issues, we propose SATS, a Separating-then-Adapting Training Strategy, which addresses OSDA-SS through two sequential steps: known/unknown separation and unknown-aware domain adaptation. By providing the model with more accurate and well-aligned unknown classes, our method ensures a balanced learning of discriminative features for both known and unknown classes, steering the model toward discovering truly unknown objects. Additionally, we present hard unknown exploration, an innovative data augmentation method that exposes the model to more challenging unknowns, strengthening its ability to capture more comprehensive understanding of target unknowns. We evaluate our method on public OSDA-SS benchmarks. Experimental results demonstrate that our method achieves a substantial advancement, with a +3.85% H-Score improvement for GTA5-to-Cityscapes and +18.64% for SYNTHIA-to-Cityscapes, outperforming previous state-of-the-art methods.

Problem

• Open-Set Domain Adaptation
• Semantic Segmentation
• Unknown Class Discovery
• Domain Adaptation
• Annotation Imbalance

Innovation

• two-stage framework
• open-set domain adaptation
• known/unknown separation
• hard unknown exploration
• semantic segmentation