Source-Free Domain Adaptation with Vision-Language Prior

2026-04-19

AI Summary

This work addresses the limitations of existing unsupervised domain adaptation methods under the challenging setting where source-domain data are unavailable and only unlabeled target-domain data can be used, a scenario in which performance is often hindered by noisy pseudo-labels or insufficient auxiliary supervision. To overcome these issues, the paper proposes DIFO++, which leverages off-the-shelf vision-language models (e.g., CLIP) as a heterogeneous prior knowledge source for the first time. By alternately optimizing mutual information between the vision-language model and the target model, and distilling multimodal knowledge into the latter, DIFO++ progressively aligns semantic representations—particularly within ambiguous “gap regions”—through a combination of class-aware attention, prediction consistency, and reference entropy minimization. Extensive experiments demonstrate that DIFO++ significantly outperforms current state-of-the-art methods across multiple benchmarks, confirming its effectiveness and robustness.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Source-Free Domain Adaptation (SFDA) seeks to adapt a source model, which is pre-trained on a supervised source domain, for a target domain, with only access to unlabeled target training data. Relying on pseudo labeling and/or auxiliary supervision, conventional methods are inevitably error-prone. To mitigate this limitation, in this work we for the first time explore the potentials of off-the-shelf vision-language (ViL) multimodal models (e.g., CLIP) with rich whilst heterogeneous knowledge. We find that directly applying the ViL model to the target domain in a zero-shot fashion is unsatisfactory, as it is not specialized for this particular task but largely generic. To make it task-specific, we propose a novel DIFO++ approach. Specifically, DIFO++ alternates between two steps during adaptation: (i) Customizing the ViL model by maximizing the mutual information with the target model in a prompt learning manner, (ii) Distilling the knowledge of this customized ViL model to the target model, centering on gap region reduction. During progressive knowledge adaptation, we first identify and focus on the gap region, where enclosed features are entangled and class-ambiguous, as it often captures richer task-specific semantics. Reliable pseudo-labels are then generated by fusing predictions from the target and ViL models, supported by a memory mechanism. Finally, gap region reduction is guided by category attention and predictive consistency for semantic alignment, complemented by referenced entropy minimization to suppress uncertainty. Extensive experiments show that DIFO++ significantly outperforms the state-of-the-art alternatives. Our code and data are available at https://github.com/tntek/DIFO-Plus.

Problem

• Source-Free Domain Adaptation
• Vision-Language Model
• Unlabeled Target Domain
• Domain Gap
• Pseudo-labeling

Innovation

• Source-Free Domain Adaptation
• Vision-Language Model
• Prompt Learning
• Knowledge Distillation
• Gap Region Reduction