Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding

2026-06-23

AI Summary

This work addresses the verification bottleneck in speculative decoding for long-context large language models, where loading the KV cache incurs substantial overhead and existing compression techniques struggle to balance accuracy and efficiency. The authors propose a sparse verification framework that uniquely integrates lookahead signals from the draft model with historical attention patterns of the target model to precisely identify critical tokens within multi-step verification windows. By computing importance scores on only a small subset of attention heads, the method drastically reduces recomputation costs. Evaluated on the Qwen2.5-72B model with 32k sequence length, the approach achieves a 27.85× speedup in self-attention computation and a 9.17× end-to-end decoding acceleration on PG-19 and LongBench benchmarks, with negligible accuracy degradation.

Abstract

While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck where Key-Value (KV) cache loading dominates latency. Existing compression methods fail in this regime: static eviction incurs accuracy loss due to saliency shift, while dynamic selection introduces prohibitive computational overhead during the verification path. We propose Dustin, a sparse verification framework designed for long-context speculative decoding. Dustin integrates lookahead signals from the draft model with historical attention from the target model to identify critical tokens with high fidelity across multi-step verification windows. To reduce recomputation latency, this approach further employs a sparse estimation scheme that restricts importance scoring to a minimal subset of attention heads. Evaluations on PG-19 and LongBench with Qwen2.5-72B demonstrate that Dustin achieves a 27.85x speedup in self-attention and a 9.17x end-to-end decoding speedup at a 32k sequence length, all with negligible accuracy degradation.

Problem

• speculative decoding
• verification bottleneck
• KV cache
• long-context LLMs
• attention sparsity

Innovation

• speculative decoding
• sparse verification
• KV cache optimization
• long-context LLMs
• attention sparsity