Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM

2026-06-28

AI Summary

This work addresses the substantial memory overhead of key-value (KV) caching in large language model inference, where existing attention sparsity–based pruning methods often incur significant performance degradation on long-context tasks. To mitigate this issue, the authors propose K-VEC, a coverage-aware KV cache eviction strategy that, for the first time, links token coverage with mutual information. K-VEC introduces cross-head and cross-layer coverage enhancement modules to prioritize retaining KV entries that maximally improve input coverage during eviction. Evaluated across 16 LongBench subsets, K-VEC achieves up to a 10.35-point improvement over state-of-the-art methods under identical eviction rates and memory constraints, effectively alleviating performance loss while maintaining a favorable balance between inference efficiency and accuracy.

Abstract

Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs. However, deploying LLMs is costly, not only due to the high computational demands of quadratic complexity of self-attention and auto-regressive generation, but also because of the significant memory overhead required for storing the key-value (KV) cache during inference. To reduce the memory cost, existing KV-cache eviction strategies leverage the sparsity in attention to selectively store a subset of tokens. While reducing the memory footprint, such approaches show a considerable drop in performance, especially in tasks that require long-context reasoning. We identify that the drop in performance is linked to a reduction in the coverage of unique tokens. Additionally, we theoretically show that reduced coverage limits the mutual information between inputs and outputs, thereby impairing predictive accuracy. To this end, we introduce K-VEC, a novel coverage-aware KV-cache eviction strategy that prioritizes token coverage while evicting tokens in the cache. K-VEC introduces a cross-head and a cross-layer coverage module to enhance token retention across attention heads and model layers, mitigating performance degradation caused by low coverage. Evaluated on 16 LongBench subsets, K-VEC exhibit up to 10.35 points improvement over the existing methods under the same eviction rate and memory constraint. Comprehensive evaluations validate the effectiveness of our approach and demonstrate its potential for efficient LLM deployment in resource-constrained settings.

Problem

• KV cache eviction
• coverage
• long-context reasoning
• memory overhead
• large language models

Innovation

• KV cache eviction
• coverage-aware
• long-context reasoning
• mutual information
• efficient LLM inference