On the Statistical Capacity of Deep Generative Models

2025-01-14

AI Summary

This work exposes a fundamental statistical limitation of deep generative models—including VAEs, GANs, and diffusion models—in modeling heavy-tailed high-dimensional data: when latent variables follow Gaussian or more general log-concave distributions, such models can only produce light-tailed samples and provably fail to universally approximate arbitrary heavy-tailed target distributions. We develop the first unified theoretical framework linking generative capacity bounds to concentration phenomena, convex geometry, the Gromov–Lévy inequality, and Riemannian probability theory under Ricci curvature constraints. Our analysis rigorously refutes the widespread assumption that “larger models and bigger datasets suffice to approximate any continuous distribution.” Through rigorous proofs and empirical validation on financial time series and synthetic heavy-tailed data, we precisely characterize the statistical capacity boundaries of mainstream architectures. This provides the first systematic, theoretically grounded criterion for model selection and architectural design in heavy-tailed regimes.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Deep generative models are routinely used in generating samples from complex, high-dimensional distributions. Despite their apparent successes, their statistical properties are not well understood. A common assumption is that with enough training data and sufficiently large neural networks, deep generative model samples will have arbitrarily small errors in sampling from any continuous target distribution. We set up a unifying framework that debunks this belief. We demonstrate that broad classes of deep generative models, including variational autoencoders and generative adversarial networks, are not universal generators. Under the predominant case of Gaussian latent variables, these models can only generate concentrated samples that exhibit light tails. Using tools from concentration of measure and convex geometry, we give analogous results for more general log-concave and strongly log-concave latent variable distributions. We extend our results to diffusion models via a reduction argument. We use the Gromov--Levy inequality to give similar guarantees when the latent variables lie on manifolds with positive Ricci curvature. These results shed light on the limited capacity of common deep generative models to handle heavy tails. We illustrate the empirical relevance of our work with simulations and financial data.

Problem

• Deep Generative Models
• Tail Features
• High-Dimensional Data

Innovation

• Deep Generative Models
• Tail Features
• Statistical Characteristics