VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

2026-05-17

AI Summary

This work addresses the challenge of inefficient long-context inference in large language models caused by the substantial memory footprint of key-value (KV) caches. While existing KV cache compression techniques improve throughput, they often introduce output deviations. To overcome this limitation, the authors propose VeriCache, a novel inference framework that achieves lossless generation under compressed KV caching. VeriCache first drafts outputs rapidly using a compressed cache and then verifies correctness with the full cache. By integrating token dropping, quantization, heterogeneous memory management, and decoupled PCIe/HBM bandwidth scheduling, it parallelizes drafting and verification. Furthermore, it exploits output similarity to extend the drafting window, reducing verification overhead. The framework guarantees bit-identical outputs compared to standard inference while achieving up to 4× higher throughput, making it particularly effective for long-sequence generation and remote prefix caching scenarios.

Abstract

The large size of the KV cache has become a major bottleneck for serving LLMs with increasing context lengths. In response, many KV cache compression methods, such as token dropping and quantization, have been proposed. However, almost all of these methods are inherently lossy-despite minimal accuracy degradation for short outputs, their outputs increasingly diverge from full-KV-cache outputs as more tokens are decoded, which leads to catastrophic failures in code generation and tool calling. We present VeriCache, the first inference framework that ensures the same output as full-KV-cache decoding but largely preserves the high decoding throughput of a range of KV cache compression algorithms. VeriCache uses the compressed KV cache to draft tokens, then verifies them against the full KV cache. While it may seem like just speculative decoding, VeriCache requires addressing a key system challenge to work-keeping the full KV cache out of GPU memory and minimizing the overhead of swapping it in for verification. The insight is two-fold: (1) compressed-KV decoding can be parallelized with full-KV swap, because one is HBM-bandwidth-bound and the other is PCIe/network-bound, and (2) the compressed KV cache often produces output similar to the full KV cache, allowing a long drafting horizon to amortize each full-KV swap. VeriCache applies to both long-context decoding and remote prefix caching, supports a broad family of token-dropping and quantization methods through a uniform compressor interface, and composes with traditional speculative decoding. Experimental results show that VeriCache achieves up to 4X higher throughput than full-KV inference while producing identical outputs.

Problem

• KV cache
• lossy compression
• LLM inference
• output divergence
• throughput bottleneck

Innovation

• KV cache compression
• lossless LLM inference
• speculative decoding
• memory hierarchy optimization
• draft-and-verify