Discrete Diffusion Models: A Unified Framework from Tokenization to Generation

2026-07-15

AI Summary

Discrete diffusion models currently lack a unified theoretical framework, hindering systematic comparison across existing approaches. This work proposes a cohesive perspective grounded in discrete state-space formulations, integrating diverse methodologies—such as transition matrices, masking/absorbing states, and score/ratio-based formulations—into a common design space. By elucidating the intrinsic connections and trade-offs among training objectives, inference algorithms, and evaluation protocols, the study clarifies the design spectrum of discrete diffusion models. Furthermore, through the unification of multiple diffusion paradigms alongside systematic optimization and scalability analyses, this research establishes a foundation for principled method comparison, guides future investigations, and fosters synergistic development across the field.

Abstract

Discrete denoising diffusion models (DDMs) have recently emerged as a compelling alternative to autoregressive (AR) modeling for discrete data, offering parallel generation and iterative global refinement capabilities. Unlike continuous diffusion, where the state space is fixed, DDMs are fundamentally shaped by how the discrete state space is constructed: the tokenization scheme, the vocabulary topology, and domain-specific structural alphabets. This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space. Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space. The framework further exposes common design trade-offs across training objectives, inference algorithms, scaling behavior, systems optimization, and evaluation protocols, suggesting several promising directions for future research.

Problem

• Discrete Diffusion Models
• Tokenization
• State Space
• Vocabulary Topology
• Structural Alphabets

Innovation

• Discrete Diffusion Models
• Unified Framework
• State Space Construction
• Tokenization
• Denoising Diffusion