AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs

2026-08-26

AI Summary

为解决累积上下文导致的推理成本增加问题，提出AsymSpec方法，通过不对称推测解码框架在保持准确性的同时加速生成。

Abstract

Agentic LLM pipelines face escalating inference costs as context accumulates across retrieval, tool use, and multi-turn interactions. To control latency, deployments routinely compress inputs, but this degrades task accuracy. Speculative decoding (SD) accelerates generation losslessly, yet it assumes the drafter and verifier share an identical context, preventing SD from resolving the accuracy-overhead trade-off. We propose AsymSpec, an asymmetric speculative decoding framework that breaks this symmetry: a lightweight drafter reads the full input while the large verifier operates on the compressed view. The drafter steers the verifier via a contrastive $δ$-fusion of logits, modulated by a divergence-aware acceptance gate that preserves verification stability and high draft acceptance rates. Evaluated across four agentic capabilities and two end-to-end agent benchmarks, AsymSpec reaches $\approx 90\%$ of full-context accuracy on average, delivering $1.3$--$1.7\times$ throughput speedups at $0.2$--$0.3\times$ the compute cost on isolated text capabilities. These results show that asymmetric context access yields substantial gains precisely when compression discards critical reasoning signals.

Problem

• Agentic LLMs
• inference costs
• context accumulation
• speculative decoding
• accuracy-overhead trade-off

Innovation

• Asymmetric Speculative Decoding
• Context-Asymmetry
• Divergence-Aware Acceptance Gate
• Agentic LLMs
• Compression