Mean-Field Parallel Decoding for Discrete Diffusion Language Models

2026-06-14

AI Summary

Discrete diffusion language models suffer from token incompatibility during parallel decoding due to independent sampling, which limits effective parallelism. This work proposes a training-free decoding framework that enables cooperative optimization of parallel token updates by introducing pairwise interactions among positions within a single forward pass, combined with variational relaxation and fixed-point iteration. For the first time, this approach achieves consistent parallel commitments via mean-field approximation without modifying the model architecture or requiring retraining. The method significantly improves the quality–latency trade-off on inference and code generation benchmarks, enabling the generation of more tokens in parallel while maintaining high output quality.

Abstract

Discrete diffusion language models enable parallel token generation, offering a pathway to low-latency decoding. However, selecting tokens independently by marginal confidence limits effective parallelism: tokens that appear reliable in isolation can form incompatible configurations when several positions are updated at once. We introduce a training-free decoding framework that coordinates these parallel updates. At each forward pass, the method assigns a commit score to each masked position and refines these scores using pairwise interactions derived from the model's predictive distributions. A variational relaxation yields a simple fixed-point update that suppresses conflicting simultaneous commitments within a single forward pass. This mechanism allows the decoder to commit more tokens in parallel while maintaining competitive generation quality. The method is lightweight, requires no auxiliary model or retraining, and drops into existing diffusion decoding pipelines without modification. Experiments on reasoning and code-generation benchmarks show consistent improvements in the quality-latency trade-off.

Problem

• discrete diffusion
• parallel decoding
• token commitment
• incompatible configurations
• language models

Innovation

• parallel decoding
• discrete diffusion
• commit coordination
• variational relaxation
• training-free