Diverse Image Priors for Black-box Data-free Knowledge Distillation

2026-04-28

AI Summary

This work addresses the challenges in black-box, data-free knowledge distillation, where limited access to original data and teacher model internals often leads to insufficient diversity in synthetic samples and weak distillation signals. To overcome these limitations, the authors propose DIP-KD, a novel end-to-end distillation framework featuring a three-stage synergistic mechanism: image prior–based diverse sample generation, contrastive learning to enhance sample discriminability, and a guided student model for soft probability distillation. For the first time, this approach systematically demonstrates the critical role of data diversity in data-free knowledge transfer. Extensive experiments across twelve benchmarks show that DIP-KD significantly outperforms existing black-box, data-free methods, achieving state-of-the-art performance.

Abstract

Knowledge distillation (KD) represents a vital mechanism to transfer expertise from complex teacher networks to efficient student models. However, in decentralized or secure AI ecosystems, privacy regulations and proprietary interests often restrict access to the teacher's interface and original datasets. These constraints define a challenging black-box data-free KD scenario where only top-1 predictions and no training data are available. While recent approaches utilize synthetic data, they still face limitations in data diversity and distillation signals. We propose Diverse Image Priors Knowledge Distillation (DIP-KD), a framework that addresses these challenges through a three-phase collaborative pipeline: (1) Synthesis of image priors to capture diverse visual patterns and semantics; (2) Contrast to enhance the collective distinction between synthetic samples via contrastive learning; and (3) Distillation via a novel primer student that enables soft-probability KD. Our evaluation across 12 benchmarks shows that DIP-KD achieves state-of-the-art performance, with ablations confirming data diversity as critical for knowledge acquisition in restricted AI environments.

Problem

• black-box knowledge distillation
• data-free
• image priors
• synthetic data
• privacy constraints

Innovation

• black-box knowledge distillation
• data-free
• image priors
• contrastive learning
• soft-probability distillation