From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning

2026-04-16

AI Summary

This work addresses key limitations of conventional token-centric speculative decoding, which suffers from error propagation in multi-step reasoning and relies on external reward models that introduce latency and hinder generalization. The authors propose SpecGuard, a novel framework that introduces, for the first time, a step-level verification mechanism based solely on internal model signals. By sampling multiple candidate reasoning steps and leveraging attention attribution alongside token-level log-probability confidence, SpecGuard constructs a lightweight internal validation signal to enable consistent and accurate step selection. This approach eliminates the need for external rewards, achieving a 3.6% accuracy improvement across multiple reasoning benchmarks while reducing latency by approximately 11%, significantly outperforming both standard speculative decoding and reward-guided methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Speculative decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose outputs that a stronger target model verifies. However, its token-centric nature allows erroneous steps to propagate. Prior approaches mitigate this using external reward models, but incur additional latency, computational overhead, and limit generalizability. We propose SpecGuard, a verification-aware speculative decoding framework that performs step-level verification using only model-internal signals. At each step, SpecGuard samples multiple draft candidates and selects the most consistent step, which is then validated using an ensemble of two lightweight model-internal signals: (i) an attention-based grounding score that measures attribution to the input and previously accepted steps, and (ii) a log-probability-based score that captures token-level confidence. These signals jointly determine whether a step is accepted or recomputed using the target, allocating compute selectively. Experiments across a range of reasoning benchmarks show that SpecGuard improves accuracy by 3.6% while reducing latency by ~11%, outperforming both SD and reward-guided SD.

Problem

• speculative decoding
• multi-step reasoning
• verification
• error propagation
• computational overhead

Innovation

• speculative decoding
• step-level verification
• model-internal signals
• attention-based grounding
• efficient reasoning