Trajectory-Level Speculative Decoding for Diffusion Language Models

2026-08-27

AI Summary

为解决扩散语言模型在低置信度下生成速度受限的问题，提出了一种基于轨迹推测的解码框架，通过构建草稿去噪轨迹并进行块级并行验证，提高了生成效率。

Abstract

Diffusion-based language models (dLLMs) enable parallel token generation through iterative denoising, but existing decoding strategies collapse to single-token generation under low confidence, severely limiting throughput. Unlike autoregressive models where speculative decoding operates on token sequences in a fixed left-to-right order, dLLMs require speculating over denoising trajectories-sequences of multi-token updates with explicit positions and unmasking orders. We develop a trajectory-level speculative framework that constructs draft denoising trajectories via confidence-stratified tree exploration and verifies them through blockwise parallel evaluation with bidirectional attention masking. Our method further introduces inter-block speculation, exploiting diffusion models' bidirectional structure to perform cross-block lookahead. We formally characterize when this approach is exact and identify trajectory drift as the fundamental cost of increased parallelism. Building on Fast-dLLM's dual-cache infrastructure, our framework reduces denoising iterations by 30-40% and increases tokens-per-step from 2.6 to 4.3, achieving 7-14x speedup over vanilla dLLMs and 1.3x over Fast-dLLM with less than 1% accuracy change across reasoning and code benchmarks.

Problem

• diffusion language models
• parallel generation
• throughput
• speculative decoding
• denoising trajectories

Innovation

• trajectory-level speculative decoding
• confidence-stratified tree exploration
• bidirectional attention masking
• inter-block speculation