Linear Convergence of Diffusion Models Under the Manifold Hypothesis

2024-10-11

arXiv.org

AI Summary

This work addresses the slow sampling convergence of diffusion models on high-dimensional data lying intrinsically on a $d$-dimensional manifold. We establish, for the first time under KL divergence, that the reverse SDE sampling converges at a rate linear in the intrinsic dimension $d$. Methodologically, we integrate score matching estimation, a novel manifold-adapted reverse SDE discretization scheme, and differential geometric analysis to rigorously characterize how manifold geometry governs error propagation. Our theoretical analysis shows that achieving $varepsilon$-accuracy requires only $O(d log(1/varepsilon))$ steps—breaking prior bounds of $O(D)$ (where $D$ is the ambient dimension) or $O(d^k)$ for $k > 1$. This bound is tight and optimal. Our result provides the first tight, linear-in-$d$ convergence guarantee for efficient sampling of diffusion models under the manifold hypothesis.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Score-matching generative models have proven successful at sampling from complex high-dimensional data distributions. In many applications, this distribution is believed to concentrate on a much lower $d$-dimensional manifold embedded into $D$-dimensional space; this is known as the manifold hypothesis. The current best-known convergence guarantees are either linear in $D$ or polynomial (superlinear) in $d$. The latter exploits a novel integration scheme for the backward SDE. We take the best of both worlds and show that the number of steps diffusion models require in order to converge in Kullback-Leibler~(KL) divergence is linear (up to logarithmic terms) in the intrinsic dimension $d$. Moreover, we show that this linear dependency is sharp.

Problem

• Analyze convergence of diffusion models under manifold hypothesis
• Improve convergence guarantees from polynomial to linear in d
• Prove sharp linear dependency on intrinsic dimension d

Innovation

• Linear convergence in intrinsic dimension d
• Novel backward SDE integration scheme
• Sharp linear dependency on d