On the Error-Correcting Effects of Stochasticity in Discrete Diffusion

2026-05-26

AI Summary

This work addresses the trade-off between sampling efficiency and sample quality in discrete diffusion models. Through a systematic analysis of the role of stochasticity in Markov transitions, the study reveals that while high randomness slows convergence, it enables error correction via symmetric state-to-state mass exchange. Building on this insight, the authors propose Discrete Churning and Restart Sampling (DCRS), an algorithm that alternately injects controlled randomness during both forward and reverse diffusion processes, guided by an information-theoretic perspective. Experiments demonstrate that DCRS reduces sampling steps by up to 10× in image generation while preserving sample fidelity, and exhibits superior performance in language generation tasks, particularly under corruption-sensitive conditions.

Abstract

Discrete diffusion models achieve strong performance in text and image generation, but their inference remains slow and must inherently balance sampling efficiency and sample quality. In this work, we present a systematic study of how the \emph{degree of stochasticity} in Markov transitions governs the sampling tradeoff. We show that highly deterministic transitions converge rapidly but suffer from error accumulation, while more stochastic transitions converge more slowly yet can achieve higher final sample quality. Using an information-theoretic analysis, we identify the underlying mechanism as an error-correcting effect induced by \emph{redundant transitions} that symmetrically exchange mass between states, and show that these transitions can provably contract sampling errors. Motivated by this analysis, we propose \emph{Discrete Churn and Restart Sampling} (DCRS), a novel inference algorithm that injects controlled stochasticity by alternating between forward and reverse diffusion processes. Experiments on synthetic datasets and large-scale benchmarks show that DCRS improves the speed-quality tradeoff in the low number of function evaluations regime. On image datasets, DCRS achieves up to a $10\times$ reduction in sampling steps compared to standard samplers while maintaining competitive sample quality, whereas on language benchmarks, we observe more nuanced behavior depending on the corruption process and sampling procedure.

Problem

• discrete diffusion
• sampling efficiency
• sample quality
• error accumulation
• stochasticity

Innovation

• discrete diffusion
• stochasticity
• error correction
• redundant transitions
• churn and restart sampling