Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding

2026-05-19

AI Summary

This work addresses the high computational and memory overhead in speculative decoding caused by dense draft trees, where existing dynamic pruning methods reduce latency at the cost of discarding useful candidates, thereby limiting acceptance rates. The authors propose Graft, a novel framework that couples pruning and retrieval as complementary operations for the first time: within the computational budget freed by dynamic depth pruning, it employs near-zero-overhead vector retrieval to compensate for pruned branches and fill topological gaps. Requiring no additional training, Graft achieves lossless acceleration, delivering an average speedup of 21.8% over EAGLE-3 on Qwen3-235B and up to 5.41× speedup on short-context tasks, while establishing new Pareto frontiers across multiple scenarios.

Abstract

Speculative decoding (SD) accelerates large language model inference by leveraging a draft-then-verify paradigm. To maximize the acceptance rate, recent methods construct expansive draft trees, which unfortunately incur severe VRAM bandwidth and computational overheads that bottleneck end-to-end speedups. While dynamic-depth pruning can reduce this latency by removing marginal branches, it also discards potentially valid candidates, preventing the acceptance rate from reaching the upper bound of dense trees. In this paper, we identify a critical opportunity in resource allocation: the transition from dense to pruned drafting frees up significant computational budget. To break this Pareto tradeoff, we introduce Graft, a compensation framework that couples pruning and retrieval as mutually reinforcing operations. Pruning supplies sufficient budget for retrieval, while retrieval compensates for pruning-induced coverage loss and recovers accepted length. By employing a sequential `prune-then-graft' mechanism, Graft attaches highly predictive retrieved tokens into positions opened by pruning, filling the topological gaps with near-zero overhead. Graft is entirely training-free and lossless. Comprehensive evaluations show that Graft establishes a new Pareto frontier across practical deployment settings, including short-context generation, long-context generation, and large-scale models. On short-context benchmarks, it achieves up to 5.41$\times$ speedup and improves average speedup over EAGLE-3 by up to 21.8% on the large-scale Qwen3-235B. We also provide a preliminary exploration of applying Graft to the DFlash-style block drafting paradigm, offering initial evidence and insights for extending grafting beyond autoregressive draft trees.

Problem

• speculative decoding
• draft tree
• acceptance rate
• computational overhead
• pruning

Innovation

• Speculative Decoding
• Prune-and-Graft
• Retrieval-Augmented Generation
• Efficient LLM Inference
• Hybrid Draft Tree