Discrete vs. Continuous Trade-offs for Generative Models

2024-12-26

AI Summary

This work addresses the fundamental trade-off between discrete-time modeling and continuous-time stochastic differential equation (SDE) modeling in denoising diffusion probabilistic models (DDPMs) and score-based generative models (SGMs). Specifically, it tackles the challenge that discretization-induced errors propagate through reverse sampling, degrading sample quality. To this end, we first unify discrete and continuous modeling paradigms by deriving a total variation (TV) distance bound—integrating discrete Girsanov transformation, Pinsker’s inequality, and the data processing inequality. This bound rigorously characterizes the performance limits of both frameworks. We further obtain an analytically tractable TV upper bound that quantitatively exposes the coupled influence of step size, noise schedule, and score estimation error. Our theoretical results provide principled, information-theoretic guidance for designing efficient and robust discrete-time sampling algorithms in diffusion models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Information Theory
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

This work explores the theoretical and practical foundations of denoising diffusion probabilistic models (DDPMs) and score-based generative models, which leverage stochastic processes and Brownian motion to model complex data distributions. These models employ forward and reverse diffusion processes defined through stochastic differential equations (SDEs) to iteratively add and remove noise, enabling high-quality data generation. By analyzing the performance bounds of these models, we demonstrate how score estimation errors propagate through the reverse process and bound the total variation distance using discrete Girsanov transformations, Pinsker's inequality, and the data processing inequality (DPI) for an information theoretic lens.

Problem

• Diffusion Models
• Discrete vs Continuous Choices
• De-noising Errors

Innovation

• Denoising Diffusion Probabilistic Models
• Score-based Generative Models
• Information-theoretic Analysis