LESS Is More: Mutual-Stability Sampling for Diffusion Language Models

2026-06-15

AI Summary

Diffusion language models typically employ a fixed number of reverse denoising steps during decoding, leading to redundant computations for already stabilized tokens and premature commitment to unstable ones, thereby compromising both efficiency and accuracy. This work proposes LESS, a training-free, model-agnostic adaptive sampling method that, for the first time, formulates token commitment as an online stopping problem. LESS introduces a dynamic mask-revocation mechanism based on mutual stability, integrating top-1 confidence, cross-step token consistency, and Jensen–Shannon divergence over the top-K predictive distribution to enable efficient and reliable token commitment. Experiments demonstrate that LESS achieves higher average accuracy than existing training-free adaptive samplers across multiple benchmarks while reducing the number of reverse steps by 72.1%, substantially lowering inference latency and computational overhead.

Abstract

Diffusion large language models (dLLMs) offer a promising alternative to autoregressive decoding by iteratively refining masked sequences, enabling parallel token updates and bidirectional conditioning. Their practical efficiency, however, is limited by sampling procedures that execute a fixed number of reverse denoising steps selected before decoding, spending computation on already-stable positions and sometimes committing unstable ones too early. We present \textsc{LESS}, a training-free, model-agnostic adaptive sampler that treats token commitment as an online stopping problem. \textsc{LESS} implements mutual-stability sampling through a joint stability rule that makes a masked position eligible for unmasking only when its top-1 prediction has high confidence, its top-1 token persists across recent reverse steps, and its predictive distribution is stable under top-$K$ inter-step Jensen--Shannon divergence. We evaluate \textsc{LESS} on Dream-7B, LLaDA-8B, and LLaDA-1.5-8B, covering full-sequence diffusion and semi-autoregressive blockwise sampling regimes, across seven benchmarks spanning general knowledge, math, and code. \textsc{LESS} improves average accuracy over strong training-free adaptive samplers while using $72.1\%$ fewer reverse steps than fixed-budget decoding. Since each reverse step requires a Transformer forward pass, these step-count reductions translate into fewer forward evaluations, lower measured wall-clock latency, and lower estimated inference compute.

Problem

• diffusion language models
• adaptive sampling
• token commitment
• reverse denoising steps
• sampling efficiency

Innovation

• mutual-stability sampling
• diffusion language models
• adaptive sampling
• Jensen-Shannon divergence
• training-free inference