SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning

2025-04-10

AI Summary

To address the high inference latency of Large Reasoning Models (LRMs) caused by long chain-of-thought reasoning and autoregressive decoding, this paper proposes a semantic-level speculative inference mechanism: lightweight models execute intermediate reasoning steps, while the base model only verifies semantically critical outputs—relaxing speculation tolerance from token-level equivalence to semantic utility preservation, thereby enabling approximate modeling of reasoning steps. The method integrates semantic-driven dynamic verification, hierarchical reasoning offloading, and coordinated scheduling with conventional speculative decoding. On multiple reasoning benchmarks, it achieves 1.5–2.5× speedup over baselines with 1.0–9.9% accuracy gains; joint deployment with speculative decoding further reduces end-to-end latency by 19.4–44.2%, significantly advancing the joint frontier of inference efficiency and accuracy.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advances in inference-time compute have significantly improved performance on complex tasks by generating long chains of thought (CoTs) using Large Reasoning Models (LRMs). However, this improved accuracy comes at the cost of high inference latency due to the length of generated reasoning sequences and the autoregressive nature of decoding. Our key insight in tackling these overheads is that LRM inference, and the reasoning that it embeds, is highly tolerant of approximations: complex tasks are typically broken down into simpler steps, each of which brings utility based on the semantic insight it provides for downstream steps rather than the exact tokens it generates. Accordingly, we introduce SpecReason, a system that automatically accelerates LRM inference by using a lightweight model to (speculatively) carry out simpler intermediate reasoning steps and reserving the costly base model only to assess (and potentially correct) the speculated outputs. Importantly, SpecReason's focus on exploiting the semantic flexibility of thinking tokens in preserving final-answer accuracy is complementary to prior speculation techniques, most notably speculative decoding, which demands token-level equivalence at each step. Across a variety of reasoning benchmarks, SpecReason achieves 1.5-2.5$ imes$ speedup over vanilla LRM inference while improving accuracy by 1.0-9.9%. Compared to speculative decoding without SpecReason, their combination yields an additional 19.4-44.2% latency reduction. We open-source SpecReason at https://github.com/ruipeterpan/specreason.

Problem

• Reduces high inference latency in Large Reasoning Models
• Accelerates LRM inference using lightweight speculative steps
• Preserves accuracy while speeding up reasoning tasks

Innovation

• Lightweight model speculates intermediate reasoning steps
• Base model assesses and corrects speculated outputs
• Semantic flexibility preserves final-answer accuracy