SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL

2026-06-17

AI Summary

This work addresses the transmission bottleneck and memory inefficiency in sparse-attention large language model inference caused by conventional RDMA-based approaches that fetch KV caches at coarse granularity. To overcome these limitations, we propose the first CXL-based fine-grained, on-demand KV caching mechanism. Leveraging CXL’s cache-line-level low-latency remote memory access capability together with a top-k dynamic selection strategy, our method loads only the necessary KV entries during inference, enabling precise and efficient data retrieval. Experimental results on DeepSeek-V3.2 demonstrate that, compared to an RDMA baseline, our approach achieves a 2.1× improvement in system throughput, reduces time-to-first-token latency by 9.7×, and decreases per-token processing time by 1.8×.

Abstract

The scaling of LLMs toward long-context inference has shifted the primary serving system bottleneck from computation to memory capacity. Traditional solutions for dense attention models rely on RDMA-based disaggregated memory pools, which perform coarse-grained fetching of the entire prefix KV cache from remote storage to local memory before decoding. However, this approach is fundamentally inefficient for emerging sparse attention models. While only a small fraction of KV entries are active during decoding, these systems still fetch the full KV cache locally, leading to severe transmission bottlenecks and local memory wastage. To address this, we propose SAC, the first efficient disaggregated KV cache system optimized for sparse attention models. By leveraging the low-latency, cache-line granularity load/store semantics of Compute Express Link (CXL), SAC fetches only the required top-k KV entries on demand during inference. Evaluations on DeepSeek-V3.2 using SGLang show that SAC achieves 2.1x higher throughput, 9.7x lower TTFT, and 1.8x lower TBT compared to RDMA-based baselines, establishing CXL-based disaggregation as the superior infrastructure for emerging sparse attention models.

Problem

• sparse attention
• KV cache
• disaggregated memory
• long-context inference
• memory bottleneck

Innovation

• Sparse Attention
• Disaggregated Memory
• CXL
• KV Cache
• LLM Inference