Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models

2026-07-06

AI Summary

This work addresses the lack of non-asymptotic convergence guarantees for stochastic gradient descent (SGD) training in score-based generative models (SGMs). Focusing on general score function parameterizations and over-parameterized two-layer ReLU networks, it provides the first non-asymptotic convergence analysis of SGD in SGM training. By integrating tools from non-convex optimization theory, the neural tangent kernel (NTK) framework, and weighted denoising score matching, the study derives explicit convergence rates for SGD under a weighted objective and establishes sharp bounds on the score approximation error along the optimization trajectory. The analysis quantifies how the choice of weighting function influences approximation accuracy, thereby offering theoretical justification for the design of weighting strategies commonly employed in practice.

Abstract

Score-based Generative Models (SGMs) have achieved impressive performance in data generation across a wide range of applications. While the statistical properties of their sampling procedures are increasingly well understood, the optimization dynamics underlying their training remain less explored. SGMs are typically trained by minimizing a weighted denoising scorematching objective, yet optimization guarantees with stochastic gradients remain limited. In this work, we study Stochastic Gradient Descent (SGD) for SGMs, contributing results in two complementary regimes. First, for general score parameterizations, we establish a non-convex convergence rate for SGD on the weighted denoising score-matching objective, with explicit dependence on the schedule-dependent weighting factors. Second, for overparameterized two-layer ReLU networks, we develop a Neural Tangent Kernel analysis tailored to diffusion training with stochastic gradients, yielding score-approximation error bounds along the SGD trajectory. Finally, our analysis quantifies the role of the reweighting factor in the score approximation error, providing theoretical guidance for weighting choices used in practice.

Problem

• Score-based Generative Models
• Stochastic Gradient Descent
• Non-asymptotic Convergence
• Denoising Score Matching
• Optimization Dynamics

Innovation

• Score-based Generative Models
• Stochastic Gradient Descent
• Non-asymptotic Convergence
• Neural Tangent Kernel
• Denoising Score Matching