SGD-KV: Summarization Guided KV Cache Compression

2026-09-02

AI Summary

为解决长文本推理中KV缓存内存瓶颈问题，提出SGD-KV框架，通过识别和优先处理擅长信息聚合的注意力头来压缩KV缓存，减少内存使用并提高性能。

Abstract

Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techniques typically rely on simple heuristics, overlooking the distinct functional roles of different attention heads. We present SGD-KV (Summarization-Guided KV Cache Compression), a head-aware framework that leverages a novel chunk-summarization diagnostic task to systematically identify and prioritize attention heads specialized in hierarchical information aggregation. Experiments on Qwen2.5-7B-1M and Qwen3-32B across diverse long-context benchmarks demonstrate that SGD-KV achieves state-of-the-art performance with contexts up to 1M tokens, while reducing KV cache memory usage by up to 75%. Our findings show that strategically allocating the KV cache budget based on the summarization score distribution of attention heads yields a superior efficiency-accuracy trade-off for long-context inference.

Problem

• large language models
• memory bottlenecks
• long-context inference
• key-value caches

Innovation

• head-aware framework
• chunk-summarization task
• KV cache compression
• long-context inference