Robust Generative Learning with Lipschitz-Regularized $alpha$-Divergences Allows Minimal Assumptions on Target Distributions

2024-05-22

AI Summary

α-divergence generative learning suffers from poor robustness and instability when the target distribution’s structure is unknown—e.g., heavy-tailed, supported on low-dimensional manifolds, or fractal sets. Method: We propose a Lipschitz-regularized α-divergence generative learning framework, integrating Wasserstein-1 metrics, group symmetry analysis, and empirical process theory to ensure convergence and training stability for both GANs and gradient flows. Contributions/Results: Theoretically, we establish, for the first time under only a weak first-moment assumption on the source distribution, the boundedness and variational differentiability of the α-divergence; derive necessary and sufficient conditions for its finiteness under heavy tails and an optimal α-selection criterion; and obtain the first sample complexity upper bound on unbounded domains. Experimentally, the framework achieves stable learning across heavy-tailed, low-dimensional, and fractal-supported distributions—without requiring prior structural knowledge.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

This paper demonstrates the robustness of Lipschitz-regularized $alpha$-divergences as objective functionals in generative modeling, showing they enable stable learning across a wide range of target distributions with minimal assumptions. We establish that these divergences remain finite under a mild condition-that the source distribution has a finite first moment-regardless of the properties of the target distribution, making them adaptable to the structure of target distributions. Furthermore, we prove the existence and finiteness of their variational derivatives, which are essential for stable training of generative models such as GANs and gradient flows. For heavy-tailed targets, we derive necessary and sufficient conditions that connect data dimension, $alpha$, and tail behavior to divergence finiteness, that also provide insights into the selection of suitable $alpha$'s. We also provide the first sample complexity bounds for empirical estimations of these divergences on unbounded domains. As a byproduct, we obtain the first sample complexity bounds for empirical estimations of these divergences and the Wasserstein-1 metric with group symmetry on unbounded domains. Numerical experiments confirm that generative models leveraging Lipschitz-regularized $alpha$-divergences can stably learn distributions in various challenging scenarios, including those with heavy tails or complex, low-dimensional, or fractal support, all without any prior knowledge of the structure of target distributions.

Problem

• Robust generative learning with minimal target distribution assumptions
• Stable training for heavy-tailed and complex distributions
• Sample complexity bounds for divergence estimation on unbounded domains

Innovation

• Lipschitz-regularized α-divergences for generative modeling
• Minimal assumptions on target distribution structure
• Stable learning across diverse distribution types