GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix

2026-08-16

AI Summary

This study addresses the challenge in multi-agent serving where unified KV cache paging fails to simultaneously preserve long prefix continuity and enable fine-grained suffix allocation. We propose an asymmetric paging mechanism that stores shared prefixes and suffixes in separate continuous hot and token-level cold pools, respectively. A dual-backend scheduler dynamically selects between compute- and storage-optimized execution paths, integrated with cascaded attention and batched GEMM operations. As the first production-grade engine implementing asymmetric granularity allocation, our approach achieves a 1.57–2.16× throughput improvement on models such as Llama-3.1-8B and up to 1.95× speedup in heterogeneous scenarios, significantly enhancing inference efficiency for multi-agent workloads.

Abstract

Production paged-serving engines apply uniform paging granularity to the KV cache, even though the two regions of a multi-agent workload have opposite storage requirements: a long shared prefix demands contiguity, while the per-request suffix demands fine-grained allocation. We present \textbf{GraniKV}, a KV-cache layer that allocates the shared prefix in a contiguous HOT pool and the suffix in a token-level COLD pool, combined with a per-step dispatcher which selects the appropriate backend among dual backends for each regime (compute-, memory-, or communication-bound). To the best of our knowledge, GraniKV is the first system to apply asymmetric paging granularity to the KV cache of a production paged-serving engine. At $L_p{=}16$\,K shared tokens GraniKV reaches $\mathbf{2.16\times}$, $\mathbf{1.98\times}$, and $\mathbf{1.57\times}$ output-token throughput over the production baseline on Llama-3.1-8B/TP=1, Qwen-2.5-14B/TP=2, and Qwen-2.5-32B/TP=4. The gain decomposes: cascade attention integration contributes the majority at saturation; the asymmetric storage layer adds $1.05$--$1.15\times$ end-to-end while being what makes the batched-GEMM prefix backend possible at all. Under heterogeneous multi-agent serving with \emph{distinct} prompts of different lengths, the attribution inverts: GraniKV sustains $\mathbf{1.95\times}$ while batch-global cascade collapses to parity --- the storage layer alone carries the win in the regime that motivates the paper.

Problem

• KV-Cache Paging
• Multi-Agent Systems
• Shared Prefix
• Asymmetric Granularity
• Paged-Serving Engine

Innovation

• Asymmetric Granularity
• KV-Cache Paging
• Multi-Agent Systems
• Shared Prefix
• Paged-Serving Engine