OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching

2026-08-08

AI Summary

This work addresses the substantial HBM memory consumption of KV caches during the decoding phase of large language models, which limits batch size and throughput. The authors propose a novel look-ahead sparse prefetching mechanism that synergistically combines attention sparsity with speculative decoding to enable on-demand loading of critical KV blocks. By leveraging tokens generated speculatively to predict future attention hotspots, the method prefetches relevant KV blocks from high-capacity memory tiers into HBM. Implemented atop vLLM, the approach incurs minimal accuracy degradation (≤0.7 points) under a 2048-token KV budget. It achieves 1.69× higher inference throughput than dense vLLM—up to 2.1× for multi-GPU long-context scenarios—and, in a prefill-decode disaggregated architecture, doubles throughput while reducing per-request KV memory by 6.5–9.7× and host memory usage on decode nodes by 2.2–2.6×.

Abstract

Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the key-value (KV) cache dominates both memory footprint and memory traffic during LLM token generation, i.e., decode. In particular, HBM capacity has become a scarce and costly resource that heavily limits inference batch size and system throughput. This paper presents OasisKV, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding. Because decode-time attention is naturally sparse, OasisKV keeps only the KV entries of the most relevant tokens in HBMs for attention computation. We observe that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD). OasisKV employs an efficient attention background pipeline to identify important KV blocks. They are then prefetched from higher-capacity memory tiers (e.g., host or remote memory) and staged in HBMs before being used in the next decode step. We implement OasisKV based on vLLM. The lookahead prediction is accurate enough to keep accuracy within 0.7 points of full attention under a 2,048-token KV budget. This lets OasisKV turn sparsity into throughput gain: $1.69\times$ over dense vLLM on the reasoning workload at 0.1 points of accuracy loss, and up to $2.1\times$ on multi-GPU long-context serving. Under prefill--decode disaggregation, OasisKV reaches about $2\times$ dense throughput while admitting each request with $6.5$--$9.7\times$ less KV and holding $2.2$-$2.6$ less decode-node host memory than full KV transfer.

Problem

• KV cache
• LLM inference
• memory bottleneck
• HBM capacity
• decode phase

Innovation

• KV cache sparsity
• lookahead prefetching
• speculative decoding
• memory disaggregation
• LLM inference optimization