AgentKV: Phase-Aware KV Eviction for Agentic LLMs

2026-09-13

AI Summary

为解决代理服务中KV缓存容量和解码带宽压力，提出AGENTKV方法，通过维护每个阶段的小查询缓冲区并评分缓存键来优化KV淘汰策略。

Abstract

Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV-eviction methods score cached keys against representative queries drawn from the most recent tokens, assuming future attention resembles recent attention. We show that agentic generation violates this assumption: future queries form a mixture over think, act, tool, and others phases, and principal-angle analysis shows these components occupy measurably different query subspaces, so recency representatives systematically undervalue keys that upcoming phases will need. We propose AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union. We further implement AGENTKV in a persistent multi-turn serving path that carries compressed KV state across turns and compacts retained KV pages online. Across two models, six task domains, and three KV budgets each, AGENTKV improves task score by 5.5 points on average over R-KV and 5.3 over Tri-attention. Relative to upstream full-KV SGLang, AGENTKV improves output-token throughput by up to 1.80x. Code: https://github.com/LiuTaowen-Tony/agentkv.

Problem

• Agentic LLMs
• KV-cache
• Attention Patterns

Innovation

• Phase-Aware
• KV Eviction
• Agentic LLMs
• Query Subspaces
• Persistent Multi-turn