Unified KV Pooling to Accelerate Long-Context LLM Serving

2026-06-10

AI Summary

This work addresses the severe inefficiency in KV cache management during long-context large language model inference, which leads to first-token latency (TTFT) as high as 30.7 seconds—far exceeding practical thresholds. To overcome this, the authors propose a unified KV pooling architecture that aggregates multiple host memory regions and SSDs into a single logical pool, complemented by a bandwidth-aware cache placement strategy. Additionally, they introduce the first KV-passthrough mechanism, enabling user-space direct access to SSDs while bypassing the kernel’s file system. This approach substantially enhances I/O parallelism and eliminates file system overhead, reducing TTFT to under 10 seconds across LLaMA-3.1-8B, GPT-OSS-20B, and Qwen3-30B-A3B models—with up to 4.1× speedup and a maximum 23.2× reduction in blocking I/O time.

Abstract

Long-context LLM serving requires offloading KV caches to host-memory and SSDs, but existing mechanisms are not designed for such long contexts. We observe significant inefficiencies in current KV caching in long contexts: high serving latency ~30.7 s, exceeding the typical TTFT requirement of 10 s by more than 3x. Our in-depth analysis explains two major reasons: (1) retrieval is serialized through host-memory and SSD, leaving other host-memory modules and SSDs underutilized, and (2) SSD-based KV retrieval spends 84% of its time in the kernel filesystem rather than actual device access. To address the problems, we propose unified KV pooling, which aggregates multiple host-memory modules and SSDs into a single logical pool and distributes KV caches across devices based on their bandwidth. To eliminate the filesystem overhead, we design KV-passthrough, which bypasses the kernel filesystem and directly accesses SSD-resident KV caches from user space via SPDK. Across evaluations on LLaMA 3.1-8B, GPT-OSS-20B, and Qwen3-30B-A3B, unified KV pooling reduces TTFT in long-contexts ~4.1x over state-of-the-art techniques, all making under 10 s. It also reduces blocked I/O time by up to 23.2x by eliminating filesystem overhead.

Problem

• long-context LLM serving
• KV caching
• serving latency
• TTFT
• offloading

Innovation

• Unified KV Pooling
• KV-passthrough
• Long-context LLM serving
• SPDK
• KV cache offloading