Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation

2025-06-09

AI Summary

Cross-domain few-shot semantic segmentation (CD-FSS) confronts two core challenges: significant domain shift and extreme scarcity of annotations in the target domain. This paper makes the novel observation that adapters inherently possess domain information decoupling capability. Leveraging this insight, we propose the structure-driven Domain Feature Navigator (DFN) decoupler—a loss-function-free paradigm that explicitly separates domain-specific and domain-invariant features. To mitigate overfitting during fine-tuning, we further introduce the SAM-SVN regularization mechanism. Our method adopts a frozen-backbone architecture with lightweight DFN adaptation, jointly optimizing structured feature decoupling and domain-invariant representation learning. Evaluated on PASCAL-5i and COCO-20i benchmarks under 1-shot and 5-shot settings, our approach achieves new state-of-the-art performance, improving mean Intersection-over-Union (mIoU) by 2.69% and 4.68%, respectively—demonstrating superior effectiveness and generalization across domains.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Segmentation
• Application Domains: Mobility, Driving & Flight

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Cross-domain few-shot segmentation (CD-FSS) is proposed to pre-train the model on a source-domain dataset with sufficient samples, and then transfer the model to target-domain datasets where only a few samples are available for efficient fine-tuning. There are majorly two challenges in this task: (1) the domain gap and (2) fine-tuning with scarce data. To solve these challenges, we revisit the adapter-based methods, and discover an intriguing insight not explored in previous works: the adapter not only helps the fine-tuning of downstream tasks but also naturally serves as a domain information decoupler. Then, we delve into this finding for an interpretation, and find the model's inherent structure could lead to a natural decoupling of domain information. Building upon this insight, we propose the Domain Feature Navigator (DFN), which is a structure-based decoupler instead of loss-based ones like current works, to capture domain-specific information, thereby directing the model's attention towards domain-agnostic knowledge. Moreover, to prevent the potential excessive overfitting of DFN during the source-domain training, we further design the SAM-SVN method to constrain DFN from learning sample-specific knowledge. On target domains, we freeze the model and fine-tune the DFN to learn target-specific knowledge specific. Extensive experiments demonstrate that our method surpasses the state-of-the-art method in CD-FSS significantly by 2.69% and 4.68% MIoU in 1-shot and 5-shot scenarios, respectively.

Problem

• Addressing domain gap in cross-domain few-shot segmentation
• Enabling efficient fine-tuning with scarce target-domain data
• Decoupling domain information without loss-based constraints

Innovation

• Adapter serves as domain information decoupler
• Propose Domain Feature Navigator for decoupling
• Use SAM-SVN to prevent excessive overfitting