Algorithm- and Data-Dependent Generalization Bounds for Score-Based Generative Models

2025-06-04

AI Summary

Existing theoretical analyses of stochastic gradient methods (SGMs) rely on coarse-grained approximations, yielding overly pessimistic generalization bounds that neglect the joint influence of optimization algorithms and data characteristics. Method: We establish, for the first time, an algorithm- and data-dependent generalization bound. Our framework incorporates algorithmic factors—including optimization hyperparameters and iteration trajectories—and data-dependent complexity measures such as Rademacher complexity. We model generalization via empirical process theory, SGD stability analysis, and fractional matching error decomposition. Results: We theoretically prove that the generalization error converges as iterations progress and is modulated by the intrinsic geometric structure of the data. Empirical validation on CIFAR-10 and CelebA demonstrates that our bound is both sensitive to hyperparameter choices and significantly tighter than classical polynomial sample-complexity bounds—achieving improved tightness and practical relevance.

Technology Category

• Machine Learning: Learning Theory
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Score-based generative models (SGMs) have emerged as one of the most popular classes of generative models. A substantial body of work now exists on the analysis of SGMs, focusing either on discretization aspects or on their statistical performance. In the latter case, bounds have been derived, under various metrics, between the true data distribution and the distribution induced by the SGM, often demonstrating polynomial convergence rates with respect to the number of training samples. However, these approaches adopt a largely approximation theory viewpoint, which tends to be overly pessimistic and relatively coarse. In particular, they fail to fully explain the empirical success of SGMs or capture the role of the optimization algorithm used in practice to train the score network. To support this observation, we first present simple experiments illustrating the concrete impact of optimization hyperparameters on the generalization ability of the generated distribution. Then, this paper aims to bridge this theoretical gap by providing the first algorithmic- and data-dependent generalization analysis for SGMs. In particular, we establish bounds that explicitly account for the optimization dynamics of the learning algorithm, offering new insights into the generalization behavior of SGMs. Our theoretical findings are supported by empirical results on several datasets.

Problem

• Analyzing generalization bounds for score-based generative models (SGMs)
• Incorporating optimization dynamics into SGM performance analysis
• Bridging theory-practice gap in SGM generalization behavior

Innovation

• Algorithm-dependent generalization bounds for SGMs
• Data-dependent analysis of score-based models
• Optimization dynamics impact on generalization