Accelerating Large Language Model Reasoning via Speculative Search

2025-05-03

AI Summary

Large language models (LLMs) suffer from substantial latency in tree-search-based reasoning due to frequent generation of intermediate thoughts, hindering practical deployment. This paper proposes Speculative Search, the first framework enabling dual-granularity speculative inference—jointly operating at both the thought level and token level. A small model generates candidate thought paths, while the large model dynamically evaluates their quality and performs rejection sampling, guaranteeing that retained thoughts meet or exceed the quality of those self-generated by the large model. We introduce a novel quality-preserving rejection mechanism that rigorously aligns thought-level fidelity with the large model’s standards. Evaluated on Qwen and Llama series models, Speculative Search achieves up to 2.12× speedup in reasoning latency, maintains reasoning quality comparable to the baseline large model, and significantly outperforms existing state-of-the-art methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Tree-search-based reasoning methods have significantly enhanced the reasoning capability of large language models (LLMs) by facilitating the exploration of multiple intermediate reasoning steps, i.e., thoughts. However, these methods suffer from substantial inference latency, as they have to generate numerous reasoning thoughts, severely limiting LLM applicability. To address this challenge, we propose a novel Speculative Search (SpecSearch) framework that significantly accelerates LLM reasoning by optimizing thought generation. Specifically, SpecSearch utilizes a small model to strategically collaborate with a large model at both thought and token levels, efficiently generating high-quality reasoning thoughts. The major pillar of SpecSearch is a novel quality-preserving rejection mechanism, which effectively filters out thoughts whose quality falls below that of the large model's outputs. Moreover, we show that SpecSearch preserves comparable reasoning quality to the large model. Experiments on both the Qwen and Llama models demonstrate that SpecSearch significantly outperforms state-of-the-art approaches, achieving up to 2.12$ imes$ speedup with comparable reasoning quality.

Problem

• Reducing inference latency in tree-search-based LLM reasoning
• Optimizing thought generation to accelerate LLM reasoning
• Maintaining reasoning quality while speeding up LLM outputs

Innovation

• Small model collaborates with large model
• Quality-preserving rejection mechanism filters thoughts
• Achieves 2.12x speedup with comparable quality