When Diffusion Model Can Ignore Dimension: An Entropy-Based Theory

2026-05-08

AI Summary

Existing convergence theories for diffusion models struggle to explain their empirical efficiency in generating high-dimensional data with only a small number of reverse steps, particularly lacking a theoretical foundation for dimension-independent sampling complexity. This work addresses this gap by introducing an information-theoretic perspective, establishing for the first time a direct link between sampling complexity and the Shannon entropy of the data distribution rather than ambient dimensionality. The authors show that discretization error is governed primarily by the entropy of latent mixture components. By analyzing Gaussian mixture models through the lens of discrete distributions and leveraging both Shannon entropy and second-order moments, they prove that the required number of sampling steps scales linearly with the latent entropy and only logarithmically with the second-order moment. This result reveals that the efficiency of diffusion models in high-dimensional, low-entropy settings stems fundamentally from compact latent representations.

Abstract

Diffusion models perform remarkably well on high-dimensional data such as images, often using only a modest number of reverse-time steps. Despite this practical success, existing convergence theory does not fully explain why such samplers remain efficient in high dimensions. Many prior KL guarantees bound the discretization error in terms of the ambient dimension, while other improved results replace this dependence using intrinsic-dimensional or geometric structure assumptions. In this work, we develop an alternative information-theoretic perspective on diffusion sampler convergence. We prove that, for Gaussian mixture targets, the discretization error is controlled by the Shannon entropy of the latent mixture component rather than by the ambient dimension. Consequently, the leading step complexity scales linearly with latent entropy and depends only logarithmically on the second moment of the data. Our analysis also extends to discrete target distributions, where the relevant complexity is the entropy of the target rather than the dimension of the embedding space. These results suggest that diffusion sampling can remain efficient in high-dimensional spaces when the data distribution admits a compact latent representation, as is widely believed to be the case for natural images.

Problem

• diffusion models
• high-dimensional data
• convergence theory
• discretization error
• Shannon entropy

Innovation

• diffusion models
• Shannon entropy
• discretization error
• latent representation
• information-theoretic analysis