$alpha$-GAN by R'{e}nyi Cross Entropy

2025-05-20

AI Summary

To address gradient vanishing and training instability in generative adversarial networks (GANs), this paper proposes α-GAN, a novel framework grounded in Rényi cross-entropy. Methodologically, it models the uncertainty of the discriminator’s soft decisions via the Rényi certainty measure and formulates a minimax adversarial objective parameterized by the Rényi order α ∈ (0,1). Theoretically, we show that α ∈ (0,1) induces an exponential amplification of gradients in probability space—effectively mitigating gradient vanishing, accelerating convergence, and recovering vanilla GAN as a special case when α = 1. Empirically, α-GAN demonstrates superior training stability and faster convergence across multiple benchmark datasets, consistently outperforming standard GANs and existing Rényi-based variants. This work extends both the theoretical foundations and practical applicability of Rényi divergence in generative modeling.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Multiagent Systems: Adversarial Agents

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper proposes $alpha$-GAN, a generative adversarial network using R'{e}nyi measures. The value function is formulated, by R'{e}nyi cross entropy, as an expected certainty measure incurred by the discriminator's soft decision as to where the sample is from, true population or the generator. The discriminator tries to maximize the R'{e}nyi certainty about sample source, while the generator wants to reduce it by injecting fake samples. This forms a min-max problem with the solution parameterized by the R'{e}nyi order $alpha$. This $alpha$-GAN reduces to vanilla GAN at $alpha = 1$, where the value function is exactly the binary cross entropy. The optimization of $alpha$-GAN is over probability (vector) space. It is shown that the gradient is exponentially enlarged when R'{e}nyi order is in the range $alpha in (0,1)$. This makes convergence faster, which is verified by experimental results. A discussion shows that choosing $alpha in (0,1)$ may be able to solve some common problems, e.g., vanishing gradient. A following observation reveals that this range has not been fully explored in the existing R'{e}nyi version GANs.

Problem

• Proposes α-GAN using Rényi measures for generative adversarial networks
• Formulates value function via Rényi cross entropy for sample source certainty
• Explores α in (0,1) to accelerate convergence and mitigate vanishing gradients

Innovation

• Uses Rényi cross entropy for GAN value function
• Parameterizes solution by Rényi order α for flexibility
• Exponentially enlarges gradient for faster convergence