CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents

2026-06-15

AI Summary

This work addresses the inefficiency of existing large language model (LLM) serving systems in handling coding agent workloads, which exhibit frequent prefix repetitions and sustained pressure on key-value (KV) cache capacity. By analyzing real-world coding assistant interaction traces, the authors propose CacheWise, a novel system that integrates prefix-aware scheduling with a lightweight, tool-call metadata–driven predictive cache eviction policy. This approach substantially enhances KV cache reuse without compromising response quality. Implemented atop vLLM, CacheWise demonstrates significant performance gains under realistic workloads, reducing KV cache evictions by 2–2.6× and accelerating overall session completion time by up to 3.5× compared to baseline systems.

Abstract

Coding agents are a fast-growing LLM application, executing as long-running closed-loop sessions in which LLM generations alternate with external tool calls. Yet, unlike chat workloads, their serving behavior has not been studied extensively. We address this gap by collecting a dataset of real-world coding assistant traces. Our analysis shows that coding agent sessions repeatedly reuse large prefixes and create sustained KVCache pressure that conventional LLM serving policies handle poorly. Based on our analysis, we present CacheWise, a KVCache management layer that improves KVCache reuse for coding agent workloads. CacheWise combines prefix-aware scheduling with reuse-aware eviction guided by lightweight predictions from tool call metadata. Implemented in vLLM and evaluated on the collected traces, CacheWise reduces KVCache evictions by up to 2-2.6x and improves total agent session completion time by up to 3.5x.

Problem

• coding agents
• KVCache management
• LLM serving
• workload analysis
• cache pressure

Innovation

• KVCache management
• coding agents
• prefix-aware scheduling
• reuse-aware eviction
• LLM serving