Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

2026-07-08

AI Summary

This work addresses the substantial memory footprint of key-value (KV) cache in large language model (LLM) inference, which significantly increases serving costs and reduces efficiency. For the first time from a systems perspective, it systematically decomposes KV cache optimization into three orthogonal dimensions: execution scheduling (temporal), placement and migration (spatial), and representation preservation (structural), thereby establishing a unified analytical framework termed sKis. This framework integrates core techniques including scheduling policies, memory management, compressed representations, and dynamic migration, revealing synergistic cross-dimensional design mechanisms and their relationships with optimization objectives. The proposed framework provides both theoretical foundations and practical guidance for building efficient LLM serving infrastructure.

Abstract

Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly. The key-value (KV) cache, which stores KV tensors during autoregressive decoding, is crucial for enabling low-latency, high-throughput LLM inference serving. In this survey, we focus on system-aware KV infrastructure for serving LLMs (abbreviated as sKis). We revisit recent work from a system behavior perspective, organizing existing efforts into three dimensions: execution and scheduling (temporal), placement and migration (spatial), and representation and retention (structural). Furthermore, we analyze cross-behavior co-design affinity and behavior-objective links, highlighting future opportunities. Our work systematizes a rapidly evolving area, providing a foundation for understanding and innovating KV cache designs in modern LLM serving infrastructure.

Problem

• large language models
• KV cache
• model serving
• memory efficiency
• inference optimization

Innovation

• system-aware KV cache
• LLM serving
• KV cache optimization
• autoregressive decoding
• memory efficiency