SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

2026-07-03

AI Summary

This work addresses the significant latency incurred by sequential tool invocation in large language model (LLM) agents, which leads to GPU underutilization. The authors propose a training-free speculative execution mechanism that forks lightweight probes early during generation to predict subsequent tool calls and executes them in parallel, thereby hiding waiting time. This approach achieves the first training-free speculation without relying on auxiliary models, historical trajectories, or static workflow graphs. By integrating confidence-based filtering and partial token acceptance, it maintains or improves task accuracy while enhancing efficiency. The system is compatible with standard APIs and orthogonal to token-level speculative decoding. Experiments show an 18% reduction in P95 latency on the GAIA benchmark using Qwen3-32B, with consistent gains across models ranging from 4B to 32B parameters and accuracy matching or exceeding the baseline.

Abstract

LLM agents are becoming a common interface for research, coding, and question answering, yet their Thought-Action-Observation loop is often serial: the model reasons, emits a tool call, then idles the GPU until the result returns. This wait consumes 16-37% of wall time in our workloads and 35-61% in prior reports. Speculative tool execution can hide this wait, but existing systems need auxiliary predictors, historical traces, or static workflow graphs, leaving a gap for training-free, day-one deployment. We observe that the model can be its own predictor: a probe forked at the start of generation predicts Qwen3-32B's upcoming tool name with 74.6-99.6% accuracy across five benchmarks. We present SPORK (Self-sPeculative fORKing), a training-free controller that dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode. A cost model captures when speculation breaks even, and each component improves one of its terms: a prefix-cache fork cuts probe cost, a confidence gate filters mispredictions, and partial-token accept turns rejected probes into speculative-decoding drafts. On acceptance, the tool result is ready when reasoning ends; on rejection, SPORK falls back to serial execution with no correctness penalty. On real-tool benchmarks, SPORK cuts Qwen3-32B's GAIA P95 by 18% (131.9 to 108.1 s); the mechanism holds across model sizes from 4B to 32B and across dense and mixture-of-experts models, with task accuracy within 1 pp of baseline or better wherever measured. SPORK deploys as a thin controller over standard completion APIs (no retraining, no auxiliary models, no offline traces) and is orthogonal to token-level speculative decoding. SPORK is open source at https://github.com/baihuajun24/spork.

Problem

• LLM agents
• tool call latency
• speculative execution
• serial inference
• GPU idle time

Innovation

• speculative execution
• tool calling
• training-free acceleration
• prefix caching
• agentic LLM