KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs

2026-02-05

AI Summary

This work addresses the high memory bandwidth overhead of KV cache in large language models during long-context reasoning, a limitation exacerbated by existing compression methods that overlook data dependency and inter-layer heterogeneity. The authors propose KV-CoRE, the first method to systematically quantify the low-rank structure of KV cache by introducing normalized effective rank as a metric for compressibility. Leveraging a gradient-free, incremental singular value decomposition (SVD), KV-CoRE enables efficient, layer-wise and sample-wise evaluation. Through extensive experiments across five English domains and 16 languages on multiple models and datasets, the study establishes the first large-scale, data-driven benchmark for KV cache compressibility, revealing a strong correlation between compressibility and performance degradation, thereby offering actionable insights for dynamic compression strategies and data-aware model design.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models rely on kv-caches to avoid redundant computation during autoregressive decoding, but as context length grows, reading and writing the cache can quickly saturate GPU memory bandwidth. Recent work has explored KV-cache compression, yet most approaches neglect the data-dependent nature of kv-caches and their variation across layers. We introduce KV-CoRE KV-cache Compressibility by Rank Evaluation), an SVD-based method for quantifying the data-dependent low-rank compressibility of kv-caches. KV-CoRE computes the optimal low-rank approximation under the Frobenius norm and, being gradient-free and incremental, enables efficient dataset-level, layer-wise evaluation. Using this method, we analyze multiple models and datasets spanning five English domains and sixteen languages, uncovering systematic patterns that link compressibility to model architecture, training data, and language coverage. As part of this analysis, we employ the Normalized Effective Rank as a metric of compressibility and show that it correlates strongly with performance degradation under compression. Our study establishes a principled evaluation framework and the first large-scale benchmark of kv-cache compressibility in LLMs, offering insights for dynamic, data-aware compression and data-centric model development.

Problem

• KV-cache
• compressibility
• data-dependent
• low-rank
• large language models

Innovation

• KV-cache compression
• low-rank approximation
• data-dependent compressibility
• Normalized Effective Rank
• SVD-based evaluation