PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving

2026-05-10

arXiv.org

AI Summary

PEEK框架通过维护增量基数树和低开销双遍历方法，优化了在线LLM服务中的缓存管理和调度问题，显著提高了缓存命中率、吞吐量等性能指标。

Abstract

We present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime. PEEK maintains an incremental radix tree over the pending queue, exposing prefix-sharing clusters no existing engine surfaces. A low-overhead dual-walk matches the tree against the engine's prefix cache to yield longest-prefix-match for every waiting request; PEEK then admits cluster pioneers first so siblings inherit the freshly cached prefix, a co-designed eviction hook protects blocks ancestral to queued demand, and a multi-lane stride scheduler bounds starvation. On SGLang and vLLM across five workloads up to 4$\times$H100 (DP=2 over TP=2), PEEK delivers up to 3.0$\times$/2.6$\times$ cache hit, 7.9$\times$/7.1$\times$ TTFT, 6.7$\times$/5.5$\times$ E2E, and 3.6$\times$/4.5$\times$ throughput gains over each engine's strongest stock baseline (SGLang/vLLM), while matching baselines within noise on workloads with no exploitable prefix structure. Wins hold as KV-cache pressure and inference parallelism scale.

Problem

• LLM serving
• cache management
• scheduling
• eviction

Innovation

• Predictive Queue-Informed
• KV Cache Management
• Incremental Radix Tree
• Longest-Prefix-Match
• Multi-lane Stride Scheduler