ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decoding

2026-04-15

AI Summary

This work addresses the high latency of multi-step tool calling, which severely hinders the deployment of large language models in real-time services. To this end, it introduces a training-free, plug-and-play acceleration method that integrates structured tool-calling patterns and retrieval-augmented mechanisms into a speculative decoding framework for the first time. The approach employs a finite-state machine to alternately fill pattern tokens and speculatively generate variable fields, while leveraging vector retrieval to reuse historical tool-call records as drafts, substantially improving generation efficiency. Experimental results demonstrate that the proposed method achieves up to 4.2× inference speedup across multiple benchmarks, significantly outperforming existing training-free speculative decoding strategies.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Tool calling has greatly expanded the practical utility of large language models (LLMs) by enabling them to interact with external applications. As LLM capabilities advance, effective tool use increasingly involves multi-step, multi-turn interactions to solve complex tasks. However, the resulting growth in tool interactions incurs substantial latency, posing a key challenge for real-time LLM serving. Through empirical analysis, we find that tool-calling traces are highly structured, conform to constrained schemas, and often exhibit recurring invocation patterns. Motivated by this, we propose ToolSpec, a schema-aware, retrieval-augmented speculative decoding method for accelerating tool calling. ToolSpec exploits predefined tool schemas to generate accurate drafts, using a finite-state machine to alternate between deterministic schema token filling and speculative generation for variable fields. In addition, ToolSpec retrieves similar historical tool invocations and reuses them as drafts to further improve efficiency. ToolSpec presents a plug-and-play solution that can be seamlessly integrated into existing LLM workflows. Experiments across multiple benchmarks demonstrate that ToolSpec achieves up to a 4.2x speedup, substantially outperforming existing training-free speculative decoding methods.

Problem

• tool calling
• latency
• large language models
• real-time serving
• multi-turn interactions

Innovation

• speculative decoding
• tool calling
• schema-aware generation
• retrieval-augmented generation
• finite-state machine