Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation

2026-09-08

AI Summary

本文通过局部信息几何方法重新思考KV缓存淘汰策略，提出基于雅可比信息容量的Jacap方法，有效提升长上下文推理性能。

Abstract

Key-value (KV) cache eviction is essential for scaling long-context inference in Large Language Models. However, existing policies predominantly rely on empirical heuristics, lacking a rigorous characterization of token utility under the inherently nonlinear softmax attention mechanism. In this work, we rethink KV cache eviction through the lens of local information geometry, modeling the attention process as a nonlinear Gaussian communication channel. By performing a first-order Taylor expansion of the attention mapping, we derive the Jacobian Information Capacity, a novel objective that explicitly captures query relevance, softmax sensitivity, and structural diversity. Guided by this theory, we introduce Jacap, a capacity-aware eviction method that utilizes softmax-aware importance weighting and statistical leverage scores for subset selection. Extensive experiments across diverse architectures and benchmarks demonstrate that \textsc{Jacap} delivers superior performance in most scenarios, particularly in high-compression regimes.

Problem

• KV cache eviction
• long-context inference
• softmax attention mechanism
• token utility

Innovation

• Jacobian Information Capacity
• Capacity-aware Eviction
• Softmax Sensitivity
• Statistical Leverage Scores