Dual-Teacher Distillation with Subnetwork Rectification for Black-Box Domain Adaptation

2026-03-24

AI Summary

This work addresses the challenges of black-box domain adaptation, where neither source data nor the source model is accessible, and only query-based predictions from the source model are available. Key difficulties include severe pseudo-label noise and underutilization of semantic priors. To tackle these issues, the authors propose a dual-teacher distillation framework combined with subnetwork correction, which jointly leverages domain-specific knowledge from the black-box source model and general semantic priors from a vision-language model to generate high-confidence pseudo-labels. Subnetwork regularization is introduced to mitigate overfitting to noisy labels, while class prototype-based self-training and prompt optimization enhance semantic consistency. Extensive experiments demonstrate that the proposed method significantly outperforms existing black-box approaches and even surpasses several white-box methods across multiple benchmarks, confirming its effectiveness and robustness.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Assuming that neither source data nor the source model is accessible, black box domain adaptation represents a highly practical yet extremely challenging setting, as transferable information is restricted to the predictions of the black box source model, which can only be queried using target samples. Existing approaches attempt to extract transferable knowledge through pseudo label refinement or by leveraging external vision language models (ViLs), but they often suffer from noisy supervision or insufficient utilization of the semantic priors provided by ViLs, which ultimately hinder adaptation performance. To overcome these limitations, we propose a dual teacher distillation with subnetwork rectification (DDSR) model that jointly exploits the specific knowledge embedded in black box source models and the general semantic information of a ViL. DDSR adaptively integrates their complementary predictions to generate reliable pseudo labels for the target domain and introduces a subnetwork driven regularization strategy to mitigate overfitting caused by noisy supervision. Furthermore, the refined target predictions iteratively enhance both the pseudo labels and ViL prompts, enabling more accurate and semantically consistent adaptation. Finally, the target model is further optimized through self training with classwise prototypes. Extensive experiments on multiple benchmark datasets validate the effectiveness of our approach, demonstrating consistent improvements over state of the art methods, including those using source data or models.

Problem

• black-box domain adaptation
• pseudo label noise
• vision-language models
• knowledge distillation
• semantic priors

Innovation

• Black-box Domain Adaptation
• Dual-Teacher Distillation
• Subnetwork Rectification
• Vision-Language Models
• Pseudo Label Refinement