DDPM Score Matching and Distribution Learning

2025-04-07

AI Summary

This work establishes theoretical connections between score estimation in Denoising Diffusion Probabilistic Models (DDPMs) and classical statistical learning—specifically, parametric estimation and density estimation. Methodologically, it provides the first proof of asymptotic efficiency for DDPM score matching; introduces a novel $(varepsilon,delta)$-PAC density estimation framework; and constructs the first general computational lower bound framework for score estimation, resolving a long-standing open problem for Gaussian mixture models via cryptographic reduction. Key contributions include: (i) achieving the minimax-optimal convergence rate for density estimation over Hölder classes; (ii) designing a quasipolynomial-time $(varepsilon,delta)$-PAC density estimator; (iii) establishing tight computational lower bounds for general Gaussian mixture models; and (iv) theoretically reproducing and substantially strengthening Song’s core result (NeurIPS ’24). The analysis unifies diffusion-based learning with foundational statistical theory, offering new insights into the statistical-computational trade-offs inherent in score-based generative modeling.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Security and Privacy: Large-scale security measurements

Abstract

Score estimation is the backbone of score-based generative models (SGMs), especially denoising diffusion probabilistic models (DDPMs). A key result in this area shows that with accurate score estimates, SGMs can efficiently generate samples from any realistic data distribution (Chen et al., ICLR'23; Lee et al., ALT'23). This distribution learning result, where the learned distribution is implicitly that of the sampler's output, does not explain how score estimation relates to classical tasks of parameter and density estimation. This paper introduces a framework that reduces score estimation to these two tasks, with various implications for statistical and computational learning theory: Parameter Estimation: Koehler et al. (ICLR'23) demonstrate that a score-matching variant is statistically inefficient for the parametric estimation of multimodal densities common in practice. In contrast, we show that under mild conditions, denoising score-matching in DDPMs is asymptotically efficient. Density Estimation: By linking generation to score estimation, we lift existing score estimation guarantees to $(epsilon,delta)$-PAC density estimation, i.e., a function approximating the target log-density within $epsilon$ on all but a $delta$-fraction of the space. We provide (i) minimax rates for density estimation over H""older classes and (ii) a quasi-polynomial PAC density estimation algorithm for the classical Gaussian location mixture model, building on and addressing an open problem from Gatmiry et al. (arXiv'24). Lower Bounds for Score Estimation: Our framework offers the first principled method to prove computational lower bounds for score estimation across general distributions. As an application, we establish cryptographic lower bounds for score estimation in general Gaussian mixture models, conceptually recovering Song's (NeurIPS'24) result and advancing his key open problem.

Problem

• Links score estimation to parameter and density estimation tasks
• Provides minimax rates for density estimation over Hölder classes
• Establishes cryptographic lower bounds for score estimation

Innovation

• DDPM score-matching achieves asymptotic efficiency
• Links generation to PAC density estimation
• Proves computational lower bounds for scores