Fast Convergence for High-Order ODE Solvers in Diffusion Probabilistic Models

2025-06-16

AI Summary

This work addresses deterministic sampling acceleration in diffusion models via probability flow ODEs, focusing on the convergence of high-order exponential Runge–Kutta solvers under finite-step budgets. We establish, for the first time under general variance schedules, a total variation (TV) error upper bound for *p*-th-order Runge–Kutta methods—explicitly decoupling score function approximation error from numerical integration error, thereby enhancing theoretical generality. Leveraging *L²* regularity of the score function—specifically bounded first- and second-order derivatives—we prove that the TV distance is controlled by *O(d^{7/4} ε_{score}^{1/2} + d(dH_{max})^p)*, where *d* is data dimension, *ε_{score}* denotes score approximation error, and *H_{max}* is the maximum step size. Empirical validation on standard benchmark datasets confirms the plausibility of our theoretical assumptions. Our analysis provides the first rigorous theoretical foundation for the accuracy–efficiency trade-off of high-order ODE solvers in diffusion sampling.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Diffusion probabilistic models generate samples by learning to reverse a noise-injection process that transforms data into noise. Reformulating this reverse process as a deterministic probability flow ordinary differential equation (ODE) enables efficient sampling using high-order solvers, often requiring only $mathcal{O}(10)$ steps. Since the score function is typically approximated by a neural network, analyzing the interaction between its regularity, approximation error, and numerical integration error is key to understanding the overall sampling accuracy. In this work, we continue our analysis of the convergence properties of the deterministic sampling methods derived from probability flow ODEs [25], focusing on $p$-th order (exponential) Runge-Kutta schemes for any integer $p geq 1$. Under the assumption that the first and second derivatives of the approximate score function are bounded, we develop $p$-th order (exponential) Runge-Kutta schemes and demonstrate that the total variation distance between the target distribution and the generated data distribution can be bounded above by egin{align*} Oigl(d^{frac{7}{4}}varepsilon_{ ext{score}}^{frac{1}{2}} +d(dH_{max})^pigr), end{align*} where $varepsilon^2_{ ext{score}}$ denotes the $L^2$ error in the score function approximation, $d$ is the data dimension and $H_{max}$ represents the maximum step size used in the solver. We numerically verify the regularity assumption on benchmark datasets, confirming that the first and second derivatives of the approximate score function remain bounded in practice. Our theoretical guarantees hold for general forward processes with arbitrary variance schedules.

Problem

• Analyzing convergence of high-order ODE solvers in diffusion models
• Bounding errors in score function approximation and numerical integration
• Developing p-th order Runge-Kutta schemes for efficient sampling

Innovation

• High-order Runge-Kutta schemes for ODEs
• Bounded score function derivatives assumption
• Total variation distance error bound