Follow the Latent Roadmap: Navigating Revocable Decoding for Diffusion LLMs with Anchor Tokens

2026-06-15

AI Summary

Diffusion-based large language models struggle to balance decoding speed and output quality in parallel generation, and existing rollback strategies are prone to error propagation and reinforcement of local mistakes. This work proposes ASRD, a training-free framework that introduces, for the first time, the concept of “anchor tokens.” By leveraging temporal consistency in the embedding space to identify reliable context, ASRD constructs a dynamic anchor cache and integrates anchor-guided generation with an orthogonal perturbation verification mechanism. This explicitly disentangles trustworthy from uncertain information, effectively breaking local error consensus. Experiments demonstrate that ASRD achieves up to a 6.4% absolute accuracy gain over state-of-the-art baselines on mathematical and code benchmarks, while accelerating inference throughput by up to 7.2×.

Abstract

Diffusion Large Language Models (dLLMs) offer a promising avenue for parallel generation but face a trade-off between decoding speed and quality. While revocable decoding strategies attempt to mitigate errors by verifying and remasking tokens, they typically operate within a mixed-quality context. This leads to two critical failures: \textit{Error Propagation}, where new tokens absorb toxic information from erroneous context, and \textit{Local Error Reinforcement}, where errors mutually reinforce each other to evade detection. To alleviate these challenges, we propose ASRD (Anchor Supervised Revocable Decoding), a training-free framework that operates within the embedding space. ASRD explicitly decouples the decoding context into trusted \textit{Anchor Tokens}, which are identified via temporal consistency, and uncertain candidates. Leveraging a dynamic Anchor Tokens Cache, we introduce two complementary mechanisms: (1) Anchor-Guided Generation, which injects entropy-weighted anchor signals into masked positions to implicitly rectify attention toward the reliable global skeleton; and (2) Anchor-Perturbed Verification, which applies orthogonal perturbations to uncertain candidate tokens, destabilizing and remasking errors driven by fragile local consensus. Extensive experiments on math and coding benchmarks demonstrate that ASRD outperforms recent remasking baselines, achieving accuracy improvements of up to 6.4\% while accelerating inference throughput by up to 7.2$\times$.

Problem

• Diffusion LLMs
• revocable decoding
• error propagation
• local error reinforcement
• parallel generation

Innovation

• revocable decoding
• anchor tokens
• diffusion LLMs
• error propagation mitigation
• training-free framework