How Query Visibility Changes KV-Cache Compression Rankings: A Matched-Budget Audit

2026-07-11

AI Summary

This work addresses a critical gap between evaluation protocols and real-world deployment in key-value (KV) cache compression: existing methods are predominantly assessed under query-aware conditions, whereas practical applications require query-agnostic compression to enable cache reuse. To bridge this disconnect, the authors propose a budget-matched auditing framework that systematically evaluates six representative compression methods against three simple baselines under both query-aware and query-agnostic protocols, using fixed models, compression ratios, and decoding settings. Leveraging the RULER-8192 and LongBench benchmarks with 50,000 bootstrap resampling trials, the study reveals for the first time that query visibility fundamentally alters method rankings, establishing an interpretable performance hierarchy correlated with the degree of query information exposure. Notably, only KeyDiff consistently outperforms the best baseline under query-agnostic conditions, while widely adopted SnapKV underperforms even a simple “keep-first-and-last-window” strategy—a gap strongly linked to its reliance on query-aware scoring.

Abstract

KV-cache compression methods are predominantly evaluated with the query appended to the context before compression -- a query-aware protocol. Yet the economic case for a compressed KV cache is reuse: compress a document once, answer many future questions against it. In that deployment, compression must happen query-agnostic -- before any question is seen. We present a matched-budget audit of six published compression methods against three trivial baselines on three open 7-9B models (144,300 paired evaluations on RULER-8192; 40,800 on LongBench; 50,000-resample paired bootstrap throughout). Everything is held fixed -- model, compression ratio, instances, decoding -- except the scoring rule. Three findings. (1) Query visibility changes the rankings: under the agnostic protocol, of the five audited methods that share a common attention backend, only KeyDiff beats a best-of-3 trivial baseline consistently (31 of 36 cells), and the most widely deployed method, SnapKV, loses to "keep the start and the recent window" on average (-0.066). (2) The per-method drop between the two protocols is ordered consistently with how visible the question is to each method's scoring signal, legible in its source code: from Delta=+0.198 for SnapKV (the question sits inside its 64-token observation window) down to Delta=+0.011 for KeyDiff (its score contains no query term at all).

Problem

• KV-cache compression
• query visibility
• compression evaluation
• query-agnostic
• model inference

Innovation

• KV-cache compression
• query-agnostic evaluation
• matched-budget audit
• attention scoring
• long-context LLMs