How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Framework

2024-10-04

arXiv.org

AI Summary

Discrete diffusion models suffer from a lack of systematic theoretical error analysis, limiting their accuracy and reliability in generative modeling. To address this, we introduce the first unified analytical framework for discrete diffusions based on Lévy-type stochastic integrals, establishing— for the first time—the stochastic integral representation of discrete diffusion processes and proposing a Girsanov-type measure transformation theorem. Building upon this foundation, we derive the first explicit KL-divergence error bound for the τ-leaping discretization scheme. Our framework bridges the theoretical gap between discrete and continuous diffusion models, explicitly characterizing error sources—including state dependence of intensity functions and approximation bias from jump truncation. It unifies and strengthens existing convergence and stability results, providing a rigorous mathematical foundation and practical design principles for developing efficient, verifiable discrete diffusion algorithms.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Discrete diffusion models have gained increasing attention for their ability to model complex distributions with tractable sampling and inference. However, the error analysis for discrete diffusion models remains less well-understood. In this work, we propose a comprehensive framework for the error analysis of discrete diffusion models based on L'evy-type stochastic integrals. By generalizing the Poisson random measure to that with a time-independent and state-dependent intensity, we rigorously establish a stochastic integral formulation of discrete diffusion models and provide the corresponding change of measure theorems that are intriguingly analogous to It^o integrals and Girsanov's theorem for their continuous counterparts. Our framework unifies and strengthens the current theoretical results on discrete diffusion models and obtains the first error bound for the $ au$-leaping scheme in KL divergence. With error sources clearly identified, our analysis gives new insight into the mathematical properties of discrete diffusion models and offers guidance for the design of efficient and accurate algorithms for real-world discrete diffusion model applications.

Problem

• Analyzes error in discrete diffusion models using stochastic integrals.
• Generalizes Poisson random measure for state-dependent intensity analysis.
• Provides first error bound for τ-leaping scheme in KL divergence.

Innovation

• Stochastic integral framework for error analysis
• Generalized Poisson random measure application
• First error bound for τ-leaping scheme