A Probabilistic Interpretation of KV Cache Eviction

2026-08-28

AI Summary

本文通过概率推理形式化KV缓存淘汰问题，提出基于期望估计的解决方法，并实现解码时修正，提高任务鲁棒性和性能。

Abstract

The premise and promise of KV (cache) eviction is simple: higher throughput can be achieved by evicting some entries from the KV cache, at a negligible cost to quality. This holds empirically for many existing methods, though most rely on creative heuristics for selecting which entries to drop. Despite recent advances, the problem of KV eviction has remained informal in the literature. This paper aims to properly formalize this problem through the lens of probabilistic reasoning and reveal what can be learned from this perspective. Concretely, we (1) formalize the problem of KV eviction and, unfortunately, prove that it is computationally hard, (2) show that by framing it probabilistically, KV eviction reduces to the problem of expectation estimation, which can be approximated through sampling, (3) show that through this probabilistic interpretation, correcting for evicted entries during decoding---a previously ignored problem---becomes feasible, and (4) reveal that existing methods in the literature are zero-variance biased estimators that can be easily adapted in order to enable decode time correction. In practice, we show that this probabilistic version of KV eviction coupled with decode time correction is more robust to different tasks compared to existing eviction methods and achieves competitive performance at the same compression budget.

Problem

• KV Cache Eviction
• Probabilistic Reasoning
• Expectation Estimation
• Decode Time Correction

Innovation

• probabilistic reasoning
• expectation estimation
• decode time correction
• zero-variance biased estimators