TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding

2026-07-03

AI Summary

Existing diffusion-based language model decoders rely solely on the current denoising snapshot to commit tokens, often leading to premature commitment due to transient high-confidence errors or delayed acceptance of valid candidates. This work proposes Trajectory-Aware Commitment Gating (TACG), a training-free decoding mechanism that introduces trajectory-aware signals for the first time. TACG integrates Temporal Implicit Logit Guidance (TILG) and Historical Gating (HG) to dynamically determine token commitment timing by comparing prediction trajectories via exponential moving averages in the natural parameter space, while remaining anchored to the base posterior distribution. Experiments demonstrate that TACG consistently improves or maintains accuracy on code and mathematical reasoning tasks across LLaDA, Dream, and LLaDA2-Mini, reduces the number of denoising steps, increases tokens generated per forward pass, and achieves these gains without increasing model complexity.

Abstract

Diffusion language models (DLLMs) generate text by iteratively denoising masked positions, exposing a trajectory of predictive distributions rather than a single instantaneous belief. Most existing decoders ignore this trajectory and commit tokens from the current snapshot alone, conflating confidence with commitment readiness: a transient top-1 peak under incomplete context can be locked in, while candidates with consistent cross-step support are delayed. We propose Trajectory-Aware Commit Gating (TACG), a training-free gate-level decoder that anchors token identities to the base posterior and uses trajectory-aware signals only to decide whether the current proposal is ready to commit. TACG combines Temporal Implicit Logits Guidance (TILG), which keeps an exponential moving average of past logits as a self-reference and contrasts the current logits against this reference in natural-parameter space, with a History Gate (HG) that enforces short-term proposal persistence before commitment. Together with a capped extra-promotion budget, these components yield a stability-constrained commit rule without auxiliary networks or extra forward passes. We evaluate TACG on LLaDA, Dream, and LLaDA2-Mini across code (HumanEval, MBPP) and math (GSM8K, MATH500) benchmarks; it typically improves or preserves accuracy while reducing denoising steps and increasing tokens per forward (TPF). The code is publicly available at https://github.com/Clarence-CV/TACG-DLLM.

Problem

• diffusion language models
• trajectory-aware decoding
• token commitment
• predictive trajectory
• denoising steps

Innovation

• Trajectory-Aware Commit Gating
• Diffusion Language Models
• Temporal Implicit Logits Guidance
• History Gate
• Decoding Efficiency