Improving Generative Adversarial Networks with Self-Distillation

2026-05-08

AI Summary

This work addresses the underutilization of exponential moving average (EMA) generators in traditional GAN training, where they are employed only during inference and thus fail to leverage their inherent stability during optimization. To close this gap, the authors propose Self-Distilled GAN (SD-GAN), which uniquely integrates the EMA generator as a teacher model that guides the student generator via perceptual loss throughout training, thereby enabling end-to-end exploitation of EMA’s stabilizing properties. Theoretical analysis demonstrates that SD-GAN achieves local asymptotic stability under the Dirac-GAN setting, effectively suppressing parasitic oscillations. Extensive experiments show consistent improvements in image generation quality—particularly in FID and random-FID—across diverse architectures and datasets, along with smoother optimization trajectories. Moreover, the method proves effective for fine-tuning pretrained GANs.

Abstract

In modern GANs, maintaining an Exponential Moving Average (EMA) of the generator's weights is a standard practice, as such an averaged model consistently outperforms the actively trained generator. However, the EMA generator is used for final deployment only and does not influence the training process. To address this missed opportunity, we introduce Self-Distilled GAN (SD-GAN) that employs the EMA generator as a teacher to guide the active generator (student) via perceptual loss. We prove the local asymptotic stability of SD-GAN in the Dirac-GAN setting and show that it dampens the parasitic cycling behavior that plagues the conventional GANs. Empirical evaluations across established architectures and datasets demonstrate that SD-GAN improves the final image quality on several metrics (FID and random-FID in particular), stabilizes the optimization trajectory and provides additional learning guidance that is not trivially correlated with the conventional adversarial loss. It also proves effective for fine-tuning pretrained GAN models.

Problem

• Generative Adversarial Networks
• Exponential Moving Average
• Self-Distillation
• Training Stability
• Perceptual Loss

Innovation

• Self-Distillation
• Exponential Moving Average (EMA)
• Perceptual Loss
• GAN Stability
• Generative Adversarial Networks