Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation

2026-08-22

AI Summary

本文提出CSWL框架，通过频率域增强和跨阶段频率重构来解决无数据知识蒸馏中生成器的偏好依赖与模式崩溃问题，提高多样性和稳定性。

Abstract

Data-Free Knowledge Distillation (DFKD) preserves privacy by transferring knowledge without real data access. However, existing generator-based DFKD methods suffer from over-reliance on teacher preferences and pattern collapse, exhibiting "generative shortcut learning" in the frequency domain: dependent on specific frequency components and frequency positions, resulting in inconsistent synthetic image quality and class diversity. In this paper, we propose a CSWL framework aimed at introducing insights from the frequency domain perspective to improve generator diversity and training stability to Close the phenomenon of Shortcut learning to Win in the Longer term. To address the issue of generative shortcut learning, we introduce frequency-domain augmentation at the feature level, encouraging the generator to attend to the full frequency spectrum and thereby suppress shortcut learning behavior. To tackle training instability, we propose a Cross-Stage Frequency Reconstruction (CSFR) auxiliary task, which implicitly constructs an Exponential Moving Average (EMA) mechanism to promote long-term optimization and stability. Extensive experiments, including downstream tasks and various image recognition datasets at multiple resolutions, validate the effectiveness of CSWL in improving both diversity and stability from the frequency view.

Problem

• Data-Free Knowledge Distillation
• generative shortcut learning
• frequency domain

Innovation

• frequency-domain augmentation
• Cross-Stage Frequency Reconstruction (CSFR)
• Exponential Moving Average (EMA)