When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need

2025-07-05

AI Summary

This paper initiates the study of robustness in data-free knowledge distillation (DFKD) against non-transferable learning (NTL) teachers—unreliable models that mislead generators toward out-of-distribution (OOD) knowledge, thereby impeding effective in-distribution (ID) knowledge transfer. To address this, we propose an adversarial trap-escaping mechanism: by evaluating the robustness of synthetically generated samples to adversarial perturbations, we distinguish and filter OOD samples—fragile samples retain ID knowledge, while robust samples trigger knowledge forgetting to eliminate OOD interference. Our method leverages GAN-based synthetic data generation, integrating adversarial analysis and robustness assessment without requiring access to real ID data. Experiments across diverse NTL teacher settings demonstrate significant improvements in both accuracy and robustness of DFKD, effectively escaping the OOD knowledge trap.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Data-free knowledge distillation (DFKD) transfers knowledge from a teacher to a student without access the real in-distribution (ID) data. Its common solution is to use a generator to synthesize fake data and use them as a substitute for real ID data. However, existing works typically assume teachers are trustworthy, leaving the robustness and security of DFKD from untrusted teachers largely unexplored. In this work, we conduct the first investigation into distilling non-transferable learning (NTL) teachers using DFKD, where the transferability from an ID domain to an out-of-distribution (OOD) domain is prohibited. We find that NTL teachers fool DFKD through divert the generator's attention from the useful ID knowledge to the misleading OOD knowledge. This hinders ID knowledge transfer but prioritizes OOD knowledge transfer. To mitigate this issue, we propose Adversarial Trap Escaping (ATEsc) to benefit DFKD by identifying and filtering out OOD-like synthetic samples. Specifically, inspired by the evidence that NTL teachers show stronger adversarial robustness on OOD samples than ID samples, we split synthetic samples into two groups according to their robustness. The fragile group is treated as ID-like data and used for normal knowledge distillation, while the robust group is seen as OOD-like data and utilized for forgetting OOD knowledge. Extensive experiments demonstrate the effectiveness of ATEsc for improving DFKD against NTL teachers. Code is released at https://github.com/tmllab/2025_ICML_ATEsc.

Problem

• Addressing data-free knowledge distillation with untrusted teachers
• Mitigating misleading out-of-distribution knowledge transfer
• Proposing adversarial trap escaping to filter OOD-like samples

Innovation

• Adversarial Trap Escaping filters OOD-like samples
• Splits synthetic data by adversarial robustness
• Forgets OOD knowledge to enhance ID transfer