RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding

2026-04-16

AI Summary

This work addresses the high inference latency of autoregressive decoding in large language models, which stems from sequential token-by-token generation. While existing training-free speculative decoding methods offer acceleration, their performance is often limited by the absence of exact matches or structural guidance. To overcome this, the paper proposes a lightweight, training-free speculative decoding framework that uniquely integrates retrieval-augmented exact matching with a context-aware logit extrapolation mechanism to generate more reliable and diverse draft sequences. Requiring no additional training, the method achieves over 2× inference speedup on SpecBench, HumanEval, and MGSM-ZH benchmarks, significantly outperforming current training-free approaches while maintaining strong generality and efficiency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Autoregressive decoding in Large Language Models (LLMs) generates one token per step, causing high inference latency. Speculative decoding (SD) mitigates this through a guess-and-verify strategy, but existing training-free variants face trade-offs: retrieval-based drafts break when no exact match exists, while logits-based drafts lack structural guidance. We propose $\textbf{RACER}$ ($\textbf{R}$etrieval-$\textbf{A}$ugmented $\textbf{C}$ont$\textbf{e}$xtual $\textbf{R}$apid Speculative Decoding), a lightweight and training-free method that integrates retrieved exact patterns with logit-driven future cues. This unification supplies both reliable anchors and flexible extrapolation, yielding richer speculative drafts. Experiments on Spec-Bench, HumanEval, and MGSM-ZH demonstrate that RACER consistently accelerates inference, achieving more than $2\times$ speedup over autoregressive decoding, and outperforms prior training-free methods, offering a scalable, plug-and-play solution for efficient LLM decoding. Our source code is available at $\href{https://github.com/hkr04/RACER}{https://github.com/hkr04/RACER}$.

Problem

• autoregressive decoding
• speculative decoding
• inference latency
• retrieval-based drafts
• logits-based drafts

Innovation

• Speculative Decoding
• Retrieval-Augmented Generation
• Training-Free Acceleration
• Autoregressive Decoding
• Efficient LLM Inference