When, What, and How: Rethinking Retrieval-Enhanced Speculative Decoding

2025-11-03

AI Summary

To address retrieval redundancy, inflexible triggering timing, and low candidate efficiency in retrieval-augmented speculative decoding, this paper proposes ReSpec—a novel framework comprising three key components: (1) an entropy-guided trigger mechanism that dynamically decides whether to retrieve based on prediction uncertainty; (2) feedback-driven parallel multi-candidate selection coupled with source-aware lenient verification, jointly leveraging retrieval outputs and model-generated confidence scores to adaptively calibrate verification strictness; and (3) an end-to-end adaptive draft-switching policy, replacing heuristic rules. Evaluated across multiple benchmarks, ReSpec achieves >33% and >25% speedup over EAGLE-2 and SAM-Decoding, respectively, while preserving output quality. It significantly improves retrieval utilization and inference efficiency, establishing new state-of-the-art performance.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Speculative decoding (SD) has emerged as an effective technique to accelerate large language model (LLM) inference without compromising output quality. However, the achievable speedup largely depends on the effectiveness of the drafting model. While model-based methods like EAGLE-2 are accurate but costly, retrieval-enhanced methods like SAM-Decoding rely on heuristic switching strategies that often trigger unnecessary retrievals. To address this, we propose ReSpec ( extbf{Re}trieval-enhanced extbf{Spe}culative Decoding), a novel framework that transforms heuristic drafter switching into adaptive decision-making. ReSpec features three core innovations: 1) An extbf{entropy-guided adaptive trigger} quantifies contextual predictability to initiate retrieval only when uncertainty is low, avoiding costly low-quality speculations. 2) A extbf{feedback-driven candidate selection} leverages historical feedback to organize multiple high-quality candidates for parallel verification, maximizing retrieval utility. 3) A source-aware extbf{relaxed verification strategy} applies strict checks to model-generated drafts while using a relaxed verification for retrieved drafts, achieving a better balance between accuracy and efficiency. Extensive experiments on Spec-Bench demonstrate that ReSpec achieves state-of-the-art acceleration,outperforming EAGLE-2 and SAM-Decoding by over $33%$ and $25%$, respectively, while maintaining output quality.

Problem

• Optimizing speculative decoding acceleration by replacing heuristic switching
• Reducing unnecessary retrievals through entropy-guided adaptive triggering
• Balancing verification strictness between model-generated and retrieved drafts

Innovation

• Entropy-guided adaptive trigger initiates retrieval at low uncertainty
• Feedback-driven candidate selection organizes multiple high-quality candidates
• Relaxed verification strategy balances accuracy and efficiency