LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization

2026-08-09

AI Summary

Existing dynamic speculative decoding methods struggle to effectively balance the benefits and costs of verification in diffusion-based draft generation, limiting inference efficiency. This work formulates dynamic speculation length selection as an expected speedup optimization problem and proposes a training-free, plug-and-play iterative algorithm that dynamically determines the optimal verification length based on a marginal gain criterion driven by the draft model’s confidence. The method introduces, for the first time, a marginal gain–driven dynamic length selection mechanism into diffusion-based draft generation and jointly optimizes parallel draft generation and verification. Experiments across six target models, three diffusion-based speculative approaches, and multiple benchmarks demonstrate consistent improvements over baselines, achieving additional speedups of 0.5–1.5× and reaching up to 8.49× the speed of autoregressive decoding.

Abstract

Speculative decoding accelerates large language model inference by drafting multiple tokens for parallel verification, with efficiency critically determined by the speculative length selected at each decoding round. Existing dynamic speculation methods select the speculation length by estimating how many tokens will be accepted, which is reasonable for autoregressive drafters that generates tokens sequentially. The recent wave of diffusion-based drafters, however, generates candidate blocks in parallel at substantially lower drafting cost, shifting the key question from how many tokens to generate to how many generated tokens are worth verifying. We therefore reformulate dynamic speculative-length selection as expected-speedup optimization and derive a marginal criterion that extends the speculative sequence only when its acceptance gain outweighs the additional verification cost. Building on this criterion, we develop \textit{LibraSpec}, a training-free and plug-and-play algorithm that iteratively determines the speculative length using drafter confidence scores. Theoretically, we prove that LibraSpec monotonically converges toward the optimal speculative length. Experiments across six target models, three diffusion-based speculative decoding methods, and math, coding, and chat benchmarks show consistent improvements under both greedy and sampling settings, achieving a further $0.5\sim1.5\times$ improvement over baselines and up to $8.49\times$ speedup over autoregressive decoding.

Problem

• speculative decoding
• diffusion-based drafter
• speculative length selection
• inference acceleration
• verification cost

Innovation

• speculative decoding
• diffusion-based drafter
• dynamic speculation length
• marginal-gain optimization
• training-free acceleration