Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms

2025-02-01

AI Summary

Discrete diffusion models suffer from low inference efficiency and poor accuracy: exact methods incur redundant computation with uncertain runtime, while τ-leaping achieves speed at the cost of only first-order accuracy. This paper proposes the first high-order numerical solver framework for discrete-state diffusion processes, introducing second-order schemes—including the θ-trapezoidal method—to discrete diffusion modeling for the first time. We theoretically establish second-order convergence in KL divergence, breaking τ-leaping’s accuracy bottleneck. Our framework integrates discrete-time Markov chain modeling, adaptive step-size control, and rigorous KL divergence error analysis. Evaluated on GPT-2–scale text generation and ImageNet–scale image synthesis, our method achieves significantly improved FID and CLIP scores over state-of-the-art approaches under identical computational budgets.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Discrete diffusion models have emerged as a powerful generative modeling framework for discrete data with successful applications spanning from text generation to image synthesis. However, their deployment faces challenges due to the high dimensionality of the state space, necessitating the development of efficient inference algorithms. Current inference approaches mainly fall into two categories: exact simulation and approximate methods such as $ au$-leaping. While exact methods suffer from unpredictable inference time and redundant function evaluations, $ au$-leaping is limited by its first-order accuracy. In this work, we advance the latter category by tailoring the first extension of high-order numerical inference schemes to discrete diffusion models, enabling larger step sizes while reducing error. We rigorously analyze the proposed schemes and establish the second-order accuracy of the $ heta$-trapezoidal method in KL divergence. Empirical evaluations on GPT-2 level text and ImageNet-level image generation tasks demonstrate that our method achieves superior sample quality compared to existing approaches under equivalent computational constraints.

Problem

• Discrete Diffusion Models
• Computational Efficiency
• Accuracy Improvement

Innovation

• Enhanced τ-leaping method
• Discrete diffusion model optimization
• High efficiency in large-scale data processing