What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation

2026-09-03

AI Summary

研究探讨了解码时KV缓存压缩中不同评分聚合方法的影响，提出基于EMA的InertiaKV及其变体，以提高解码吞吐量并保持排名稳定性。

Abstract

Decoding-time KV cache compression research focuses heavily on designing better token scoring functions, while the temporal rule that aggregates scores across decode steps is often treated as an implementation detail. Under aggressive KV compression, we find that exponential-moving-average (EMA) aggregation makes approximately order-preserving scorer modifications largely indistinguishable at the eviction-set level. Value-norm and entropy variants remain highly correlated with attention and produce nearly unchanged retention sets, whereas KeyDiff, key norm, recency, and a learned scorer alter the ranking and degrade substantially. We associate this stability with the evaluated aggregation, which couples layer weighting and temporal retention. Building on this observation, we introduce InertiaKV, an EMA-based decoding-time eviction method, and InertiaKV-Lazy, its periodic-refresh variant, which yields 1.34-1.46x decode throughput relative to full refresh InertiaKV. We also study Score-Free decoding as a separate empirical operating point: it scores the full context once at the first decode step, freezes that ranking, and incurs an average quality change of +0.03 while removing all subsequent scoring. Across six open-weight backbones and the LongBench, LongBench-v2, and RULER benchmarks, the results identify temporal aggregation and ranking preservation as distinct, consequential design factors; they do not imply that scoring quality is irrelevant in general.

Problem

• KV cache compression
• temporal aggregation
• ranking preservation
• token scoring functions
• exponential-moving-average

Innovation

• Exponential Moving Average (EMA)
• Temporal Aggregation
• Ranking Preservation
• InertiaKV
• Score-Free Decoding