A Random Matrix Theory Perspective on the Consistency of Diffusion Models

2026-02-02

AI Summary

This work investigates the fundamental cause of high sample consistency in diffusion models trained on limited data. By constructing an analytical framework for linear diffusion models grounded in random matrix theory, the study establishes a novel connection between spectral analysis and diffusion consistency. It introduces a noise-level renormalization mechanism and extends the deterministic equivalent approach to fractional matrix powers to characterize the full sampling trajectory. The resulting theory accurately predicts linear diffusion dynamics and reveals systematic bias patterns across data subsets in the non-memorizing regimes of UNet and DiT architectures. These findings quantitatively link generation stability to the spectral properties of the training data.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Diffusion models trained on different, non-overlapping subsets of a dataset often produce strikingly similar outputs when given the same noise seed. We trace this consistency to a simple linear effect: the shared Gaussian statistics across splits already predict much of the generated images. To formalize this, we develop a random matrix theory (RMT) framework that quantifies how finite datasets shape the expectation and variance of the learned denoiser and sampling map in the linear setting. For expectations, sampling variability acts as a renormalization of the noise level through a self-consistent relation $\sigma^2 \mapsto \kappa(\sigma^2)$, explaining why limited data overshrink low-variance directions and pull samples toward the dataset mean. For fluctuations, our variance formulas reveal three key factors behind cross-split disagreement: \textit{anisotropy} across eigenmodes, \textit{inhomogeneity} across inputs, and overall scaling with dataset size. Extending deterministic-equivalence tools to fractional matrix powers further allows us to analyze entire sampling trajectories. The theory sharply predicts the behavior of linear diffusion models, and we validate its predictions on UNet and DiT architectures in their non-memorization regime, identifying where and how samples deviates across training data split. This provides a principled baseline for reproducibility in diffusion training, linking spectral properties of data to the stability of generative outputs.

Problem

• diffusion models
• consistency
• random matrix theory
• reproducibility
• data splits

Innovation

• Random Matrix Theory
• Diffusion Models
• Deterministic Equivalence
• Spectral Anisotropy
• Sampling Consistency