CachePrune: Privacy-Aware and Fine-Grained KV Cache Sharing for Efficient LLM Inference

2026-05-22

AI Summary

This work addresses the privacy risks inherent in cross-user KV cache sharing during large language model inference, where existing defenses that disable sharing incur substantial efficiency penalties. To reconcile privacy and performance, the authors propose a fine-grained, token-level privacy-aware KV cache sharing mechanism integrated into the vLLM framework. By leveraging sensitive-content masking, variable-length cache segmentation, and efficient retrieval, the method dynamically reuses only non-sensitive cache segments. This approach achieves the first token-level secure cache reuse, overcoming the limitations of conventional coarse-grained strategies. Evaluated across three datasets, it completely eliminates side-channel leakage through KV caches while reducing time-to-first-byte latency by 4.5× and improving cache hit rates by 44% compared to the state-of-the-art baseline.

Abstract

Large Language Models (LLMs) rely on Key-Value (KV) caching to accelerate inference, and many serving systems further share the KV cache across users' requests to reduce redundant computation. While widely adopted, unrestricted cross-user sharing introduces side-channel vulnerabilities, allowing an adversary to infer user inputs by probing for cache reuse. Existing defenses disable sharing entirely to prevent leakage; yet such a coarse-grained strategy sacrifices substantial reuse potential, since prompts often include large portions of privacy-irrelevant segments, such as system instructions or publicly accessible materials. Building on this, we present CachePrune, a privacy-aware KV cache sharing mechanism that enables fine-grained reuse of KV entries across requests. Realizing such fine granularity requires token-level cache management, as reusable segments vary in length and position due to sensitivity masking, making reuse more complex than the fixed-size or sentence-level chunking used in existing coarse-grained schemes. Specifically, CachePrune makes fine-grained reuse practical by addressing two key challenges: accurately and efficiently deriving reusable KV segments and efficiently retrieving them over variable-length spans. We implement CachePrune on top of vLLM and evaluate it on three datasets, showing that it eliminates direct leakage through KV cache reuse side channels while reducing TTFT by 4.5x and increasing cache hit rates by 44% compared with state-of-the-art approaches.

Problem

• KV cache sharing
• privacy leakage
• side-channel vulnerability
• fine-grained reuse
• LLM inference

Innovation

• KV cache sharing
• privacy-aware inference
• fine-grained caching
• side-channel mitigation
• LLM serving