Analysis of Diffusion Models for Manifold Data

2025-02-01

AI Summary

This work investigates the reverse-time dynamics of generative diffusion models on low-dimensional Gaussian mixture manifolds embedded in high-dimensional ambient space, focusing on characterizing the onset times of two phase transitions: “species differentiation” and “collapse.” We derive an exact mutual information (free energy) formula based on a generalized linear model and perform asymptotic analysis in the joint large-dimension and exponential-sample-size limit. This yields the first closed-form analytical expressions for phase transition times as functions of key geometric parameters—including the ratio of manifold dimension to ambient dimension. Our results establish a quantitative link between manifold geometry and diffusion dynamics, revealing the critical mechanism underlying the implicit modeling capability of diffusion models on nontrivial manifolds. The theory provides a foundational understanding of training and sampling dynamics for diffusion models applied to real-world low-dimensional data.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Diffusion Models for Vision
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

We analyze the time reversed dynamics of generative diffusion models. If the exact empirical score function is used in a regime of large dimension and exponentially large number of samples, these models are known to undergo transitions between distinct dynamical regimes. We extend this analysis and compute the transitions for an analytically tractable manifold model where the statistical model for the data is a mixture of lower dimensional Gaussians embedded in higher dimensional space. We compute the so-called speciation and collapse transition times, as a function of the ratio of manifold-to-ambient space dimensions, and other characteristics of the data model. An important tool used in our analysis is the exact formula for the mutual information (or free energy) of Generalized Linear Models.

Problem

• Analyze time reversed dynamics of diffusion models
• Compute transitions in manifold data models
• Determine speciation and collapse transition times

Innovation

• Generative diffusion models analysis
• Manifold data statistical modeling
• Mutual information exact formula