SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharing

2026-06-14

AI Summary

This work addresses the rapid growth of KV cache memory consumption in multi-turn conversations, which often exceeds GPU HBM capacity and necessitates offloading to CPU or SSD—introducing high latency and limiting context length. To overcome this bottleneck, the authors propose a cooperative KV cache management mechanism under heterogeneous model colocation, leveraging NVLink to efficiently share idle GPU memory across devices. The approach retains only the KV cache of currently active layers locally and dynamically loads caches on a per-layer basis, minimizing interference with colocated models. Experimental results demonstrate that, compared to vLLM and SGLang, the method reduces P99 time-to-first-token latency by up to 69% and extends the maximum supported context length by up to 3.98×.

Abstract

Multi-turn conversation is a fundamental scenario in LLM applications, widely used in chatbots and AI agents. As the conversation evolves, historical tokens accumulate continuously. Existing systems cache their key-value (KV) pairs to avoid redundant computation. However, limited GPU memory (HBM) capacity often forces these KV caches to be offloaded to CPU memory or SSD, making KV cache reloads increasingly costly in terms of latency as the context grows. Meanwhile, the constrained HBM capacity also limits the maximum inference length, thereby restricting the number of turns that can be supported in a conversation. To address these two challenges, we propose SwiftCache, a collaborative inference system that enables heterogeneous models to share underutilized GPU memory and NVLink bandwidth within a server. Specifically, models with low KV cache demand donate idle GPU memory to store the prefix cache of high-demand models, allowing cross-model KV cache sharing over NVLink and avoiding slow PCIe transfers. SwiftCache further reduces memory pressure by keeping only the KV cache of the currently active layer in local GPU memory, thereby enabling longer-context inference. Our experiments on real-world workloads show that SwiftCache reduces P99 time-to-first-token (TTFT) by up to 69% and extends maximum context length by up to 3.98x compared to vLLM and SGLang, with minimal interference to co-located models.

Problem

• multi-turn conversation
• KV cache
• GPU memory limitation
• long-context inference
• LLM serving

Innovation

• KV cache sharing
• heterogeneous LLM serving
• multi-turn conversation
• GPU memory pooling
• NVLink