Recency/Frequency Adaptive KV Caching for Large Language Model Serving

2026-06-19

AI Summary

This work addresses the inefficiency of conventional LRU-based KV cache management under diverse inference workloads, where inter-task eviction degrades cache performance. To mitigate this, the study introduces a frequency-aware mechanism into KV cache management and proposes an adaptive algorithm that dynamically allocates cache space by jointly considering recency and frequency of KV blocks. Integrated into the vLLM inference framework, the method significantly enhances cache hit rates and inference efficiency: on synthetic document QA tasks, it achieves up to a 10.8% improvement in cache hit rate and a 12.6% reduction in time-to-first-token latency; on real-world dialogue tasks, it yields gains of 2.1% and 2.0%, respectively. The approach also demonstrates strong interpretability and scales effectively with batched inference.

Abstract

Key-value (KV) caching is a powerful technique for accelerating large language model inference and generation. Inference workloads are large and diverse, which makes them difficult to cache effectively. Existing cache management strategies adopt the least-recently-used policy for evicting cache blocks. However, LRU leads to multiple unrelated workloads flushing each other's caches. To address this, we integrate adaptive caching that dynamically allocates cache space between recently and frequently occurring KV blocks. Evaluations show that it improves the KV cache hit rate by up to 10.8% and reduces time to first token by up to 12.6% over naive vLLM on synthetic document question answering workloads, and 2.1% and 2.0% respectively on real-world conversation workloads. The method generalizes well to batch inference and demonstrates clear interpretability while effectively accommodating diverse workloads.

Problem

• KV caching
• cache management
• large language models
• inference workloads
• recency/frequency

Innovation

• adaptive KV caching
• recency/frequency trade-off
• LLM inference optimization
• cache hit rate
• time to first token