Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches

2026-05-12

AI Summary

This work addresses the inefficiency of existing LLM prefix cache eviction policies—such as LRU—that disregard the varying reuse rates across tokens of different semantic types, leading to suboptimal GPU memory utilization. To overcome this limitation, the authors propose SAECache, a semantic-aware adaptive prefix caching mechanism that dynamically optimizes KV cache management through a task-specific multi-queue architecture, an online feedback–driven semantic token weighting scheme, and a fully parameterized self-adaptive learning framework that requires no manual tuning. Experimental results demonstrate that under heterogeneous workloads, SAECache improves time-to-first-token (TTFT) latency by 1.4–2.7× over production-grade baselines and effectively mitigates the severe performance degradation—up to 2.7×—observed with fixed eviction strategies when workload characteristics mismatch policy assumptions.

Abstract

Prefix caching is a key optimization in Large Language Model (LLM) serving, reusing attention Key-Value (KV) states across requests with shared prompt prefixes to reduce expensive prefill computation. However, its benefit depends critically on the eviction policy as GPU memory is scarce, and existing policies such as LRU largely treat cached blocks uniformly. This view ignores a fundamental property of LLM prompts: not all tokens are equally worth caching. We show that different token types within a prompt, including system prompts, user queries, tool outputs, model responses, and chain-of-thought reasoning, exhibit up to 756x variation in reuse rates, yet no existing eviction policy exploits this signal. In this paper, we present SAECache (Semantic-Adaptive Eviction for prefix caches), a semantic-adaptive prefix cache eviction policy that addresses this gap through three innovations: (1) a multi-queue architecture that routes KV blocks to task-specific queues with tailored priority metrics, capturing both session reuse in multi-turn requests and structural reuse in templated single-turn requests; (2) a semantic-aware token weighting mechanism that learns the reuse value of different token types online through eviction feedback; and (3) a fully adaptive online learning schema for all parameter updates, including log-normal timing parameters, position decay power, queue weights, and meta-parameters, which eliminates manual tuning and enables automatic adaptation to deployment-specific workload characteristics. Through extensive evaluation across heterogeneous workloads, we demonstrate that SAECache achieves 1.4x-2.7x TTFT improvement over production-style baselines, while fixed-parameter alternatives can degrade by up to 2.7x under workload mismatch -- a failure mode our adaptive approach avoids entirely.

Problem

• prefix caching
• eviction policy
• LLM serving
• token reuse
• KV cache

Innovation

• prefix caching
• semantic-aware eviction
• online learning
• LLM serving
• KV cache optimization