RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation

2026-08-09

AI Summary

This work addresses the challenge of KV cache memory constraints in long-context large language model inference, where existing methods struggle to allocate limited cache budgets effectively across layers. The authors propose a dynamic cache allocation strategy grounded in perturbation propagation effects: by injecting norm-adaptive perturbations into value caches and measuring their impact on the final output distribution via KL divergence, they quantify each layer’s sensitivity to compression. This yields a normalized sensitivity profile, which is then mapped exponentially to distribute the cache budget across layers. Notably, this approach is the first to assess inter-layer sensitivity based on predictive perturbations, overcoming limitations of methods relying on layer depth or static attention statistics. Evaluated on the LongBench benchmark, it significantly outperforms current KV compression techniques under identical cache budgets, achieving state-of-the-art average performance.

Abstract

Long-context LLM inference is bottlenecked by KV cache memory, yet distributing a limited cache budget across layers remains challenging. Existing methods rely on proxies such as layer depth, attention statistics, or representation change. These proxies do not measure how perturbations at each layer propagate to the output and may therefore cause sensitive layers to be underallocated while tolerant layers are overallocated. To address this issue, we propose RippleKV, which allocates cache across layers by estimating how perturbations to each layer's value cache affect the final predictive distribution. RippleKV independently injects norm-adaptive perturbations into each layer's value cache and measures the induced KL divergence at the model output over a small calibration set. Averaging these responses yields a sensitivity profile specific to the model that need not vary monotonically with depth. RippleKV then converts the sensitivity profile into layer budget multipliers by normalizing the sensitivity scores and applying an exponential mapping. A ratio parameter controls the allocation disparity between sensitive and tolerant layers, while a final normalization preserves the KV cache budget. Experiments on LongBench demonstrate that RippleKV achieves the highest average performance among the evaluated KV cache compression methods under matched cache budgets.

Problem

• KV cache allocation
• long-context LLM inference
• layer sensitivity
• memory bottleneck
• cache budget distribution

Innovation

• KV cache allocation
• perturbation propagation
• sensitivity profiling
• long-context LLM inference
• KL divergence