Epiphany-Aware KV Cache Eviction Without the Attention Matrix

2026-06-24

AI Summary

This work addresses the bottleneck posed by KV caching in long-context inference, where existing attention-weight-based eviction methods offer limited effectiveness and poor compatibility with efficient inference kernels. The authors propose EpiKV, which introduces the epiphany score—a novel token importance metric derived from changes in internal model representations during forward propagation. This approach requires neither explicit attention matrix computation, additional training, nor custom kernels, enabling seamless integration with standard inference stacks such as FlashAttention. By combining causal rolling z-score detrending with a cache eviction strategy, EpiKV achieves 72% accuracy on MATH-500 with a 4096-token cache and 37% on AIME-2024 with an 8192-token cache, while accelerating inference by up to 2.8× and extending effective context length by up to 16×.

Abstract

As reasoning models emit chains of thought tens of thousands of tokens long, KV cache increasingly becomes a deployment bottleneck. Existing cache eviction methods rank tokens by attention weight, which is a noisy importance proxy in long reasoning traces, and prohibits the use of fused kernels in production inference by forcing the model to materialize the attention matrix. In this work, we instead score tokens with a metric we term the epiphany score: the change in the model's internal representation, read directly from the forward pass with no attention matrix and negligible extra state. Our resulting cache eviction method, EpiKV, requires no training, classifier, or custom kernel, and can be used directly in FlashAttention inference stacks unchanged -- scaling to a 16x longer feasible context than attention-based scoring. upper-mid layers negatively) and remove a positional trend with a causal rolling z-score. At a 4096-token cache EpiKV reaches 72% on MATH-500, matching the strongest attention-based baseline (ThinKV 71%, H2O 67%); a lag-normalized KV variant reaches 37% on AIME-2024 at 8192 tokens against the best of them (33%), at up to 2.8x the speed.

Problem

• KV cache eviction
• attention matrix
• long reasoning chains
• inference efficiency
• token importance

Innovation

• KV cache eviction
• epiphany score
• attention-free
• long-context reasoning
• FlashAttention