Speculate with Memory: Lossless Acceleration for LLM Agents

2026-07-13

AI Summary

This work addresses a critical limitation in existing speculative execution methods for large language model (LLM) agents—the absence of a memory mechanism that enables continuous improvement of prediction quality through historical experience. To this end, the paper introduces the first online memory system for speculative execution, integrating a contrastive transition table, episodic memory, and a confusion tracker. By synergistically combining contrastive learning, episodic retrieval, and error suppression, the framework leverages lightweight models during idle periods to perform zero-latency speculation. Empirical results demonstrate substantial gains: action prediction accuracy improves by 19–39% relative across six benchmarks, while observation prediction achieves up to 2.5× speedup. Notably, performance consistently enhances as memory accumulates, and the approach remains effective across speculative models of varying computational costs.

Abstract

Speculative execution accelerates LLM agents by using a smaller, cheaper model to predict and pre-launch the next step while the environment is idle. However, existing speculators are stateless and discard all information between tasks, preventing prediction quality from improving with experience. We equip the speculator with three online memory systems that learn from past agent trajectories: a contrastive transition table tracking action-sequence statistics, an episodic memory retrieving contextually similar segments, and a confusion tracker suppressing recurring errors. We evaluate this approach on six benchmarks spanning three speculation types: action prediction, observation prediction, and chained prediction. Memory-augmented speculation yields a 19--39\% relative accuracy improvement on action prediction and up to a $2.5\times$ increase on observation prediction tasks with repetitive action spaces. These gains grow continuously as memory accumulates and generalize across speculator models of varying cost. All speculation is lossless because it runs during idle time at zero added wall-clock cost, and the actor's trajectory is identical to non-speculative execution.

Problem

• speculative execution
• LLM agents
• stateless speculator
• prediction quality
• memory

Innovation

• speculative execution
• memory-augmented LLM agents
• online memory systems
• lossless acceleration
• trajectory learning