Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction

2026-05-10

AI Summary

This work addresses the substantial memory and computational overhead imposed by KV caching in long-context reasoning, where existing pruning methods often degrade model performance. The authors propose a learnable KV cache pruning approach featuring a global retention mechanism, reframing pruning not merely as compression but as a means to enhance inference quality. By employing a lightweight retention gate to predict the future utility of tokens—augmented with query-agnostic geometric retention proxies, shared scoring projections, and a global competition strategy across layers, heads, and modalities—the method effectively mitigates attention dilution under a unified memory budget. Experiments demonstrate that the approach significantly reduces KV cache consumption while matching or even surpassing the performance of full caching across diverse benchmarks, including long-context language modeling, vision-language tasks, and multi-turn dialogue.

Abstract

The key-value (KV) cache is a major bottleneck in long-context inference, where memory and computation grow with sequence length. Existing KV eviction methods reduce this cost but typically degrade performance relative to full-cache inference. Our key insight is that full-cache attention is not always optimal: in long contexts, irrelevant tokens can dilute attention away from useful evidence, so selective, learnable eviction can improve generation rather than merely approximate the full cache. We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget. Lightweight retention gates assign utility scores to cached KV entries, and a shared final scoring projection calibrates these scores across all layers and heads. This enables a single global eviction policy in which tokens from different layers, heads, and modalities compete directly for cache capacity. We further provide theoretical analysis showing that preferentially retaining useful tokens reduces attention dilution, and we justify geometric retention as a query-agnostic proxy for future utility. Across diverse long-context language and vision-language reasoning, and multi-turn dialogue benchmarks, our method substantially reduces KV memory while matching or surpassing full-cache inference. These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.

Problem

• KV cache
• long-context inference
• attention dilution
• memory bottleneck
• token utility

Innovation

• KV cache eviction
• long-context reasoning
• learnable retention
• attention dilution
• global calibration