Understanding Parallel Samplers in Masked Diffusion via Random Walks on Graphs

2026-06-22

AI Summary

This work addresses the lack of systematic understanding regarding the effectiveness and applicability of parallel sampling strategies in masked diffusion models. The authors propose a controllable and verifiable benchmark framework grounded in random walks on graphs, theoretically demonstrating that the performance of parallel sampling is highly dependent on the underlying graph structure. Leveraging this insight, they design a logarithmic-step, theoretically exact bisection sampler. Experiments across diverse graph topologies reveal that the relative merits of different sampling strategies are significantly influenced by graph connectivity. When applied to an OpenWebText-pretrained model, the proposed bisection sampler markedly improves sampling efficiency while preserving generation quality, thereby achieving a better trade-off between speed and fidelity.

Abstract

In this paper, we propose using random walks on graphs as a verifiable sandbox to study different parallel sampling strategies in masked diffusion models (MDMs). We train an MDM on random walk samples from a fixed graph. The graph or the transition kernel is never shown to the model explicitly and plays the role of latent structure in the sequences, albeit one that is controllable and can be used for quantitative evaluation. Thus, this framework enjoys a Sudoku-like validity check: verifying that an output is a valid walk and estimating the Markov kernel from the walks to measure distribution fidelity. Using simple graphs, we theoretically prove that parallel unmasking via widely used scores like lowest entropy is not uniformly better than a random parallel sampler; the performance critically depends on the structure of the underlying graph. We develop a new bisection sampler for random walks, which takes logarithmic steps in the sequence length and is provably exact under perfect training. Experiments on various graph walk tasks show that different parallel samplers are better for different graphs even in practice. Our initial experiments on a pretrained OpenWebText MDM show that the bisection-style samplers improve speed-quality tradeoffs even for language generation. Together, these results position graph random walks as a mechanistic benchmark for diagnosing and designing parallel samplers for masked diffusion models.

Problem

• masked diffusion models
• parallel sampling
• random walks
• graph structures
• sampling strategies

Innovation

• masked diffusion models
• parallel sampling
• random walks on graphs
• bisection sampler
• distribution fidelity