An exact information theory of generalization phase transitions in Bayesian diffusion models

2026-07-08

AI Summary

This work uncovers the fundamental mechanism by which diffusion models avoid the curse of dimensionality and achieve generalization in high-dimensional spaces. To this end, the authors propose the Bayesian Information-Restricted Diffusion (BIRD) framework, which enables time reversal of the diffusion process by inferring the training data source of each pixel through Bayesian posterior inference based on information-constrained observations of noise. Theoretically, they establish a phase transition boundary between generalization and memorization, governed jointly by training data size, generation time, and information constraints, and prove that the generative process evolves precisely along this boundary. Experiments across multiple datasets validate these predictions: both localized BIRD models and early-trained UNet/DiT architectures closely adhere to the boundary, highlighting the critical role of information constraints in overcoming the curse of dimensionality.

Abstract

How diffusion models circumvent the curse of dimensionality to learn complex distributions over high dimensional spaces from a finite training set, instead of memorizing it, remains a fundamental mystery. To address this, we introduce analytically tractable Bayesian information restricted diffusion (BIRD) models, in which each pixel observes restricted information about noisy data. A BIRD model time-reverses diffusion by inferring which past training sample produced its current restricted observation using the Bayesian posterior. This model class generalizes existing analytical diffusion models that use spatially local information restriction. We show that spatially local BIRD models closely approximate trained diffusion models \textit{early in training}, across different architectures such as UNets and DiTs. Under minimal assumptions on the data distribution, we identify an information-theoretic phase boundary between memorization and generalization in the joint space of amount of training data, time in the reverse generative process, and amount of information restriction: a BIRD model memorizes when the mutual information between its restricted noisy observations and the training data exceeds the log number of training points, and it generalizes otherwise. Experiments across a range of datasets confirm our theoretically predicted location for the transition. We find that generation proceeds near the edge of memorization: both spatially local BIRD models and early-training diffusion models track the memorization-generalization phase boundary by increasingly restricting information over time. Overall, our results reveal a fundamental role for information restriction in generative AI to circumvent the curse of dimensionality.

Problem

• generalization
• memorization
• diffusion models
• curse of dimensionality
• information restriction

Innovation

• Bayesian diffusion models
• information restriction
• generalization phase transition
• mutual information
• curse of dimensionality