SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

2026-06-30

AI Summary

This work addresses the high memory overhead of key-value (KV) caching in long-context large language model (LLM) inference, where existing compression methods struggle to balance efficiency with contextual fidelity. The authors propose SeKV, a resolution-adaptive semantic KV caching mechanism that partitions input context into semantically coherent segments guided by entropy. Without modifying the original LLM architecture, SeKV leverages a GPU-CPU hierarchical storage system and lightweight summary vectors, employing low-rank SVD bases combined with a learnable query-aware expansion module to enable fine-grained, on-demand reconstruction of relevant context. Experimental results demonstrate that under 128K-token contexts, SeKV reduces GPU memory consumption by 53.3% while achieving an average performance gain of 5.9% over the strongest semantic compression baseline.

Abstract

Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token-level detail from a compressed span once it becomes relevant during generation. As a solution, we propose SeKV, a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans and stores them across a GPU-CPU memory hierarchy without discarding information. Each span keeps a lightweight summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction. A trained zoom-in mechanism selectively expands query-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU. SeKV enables adaptive token-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0.05% trainable parameters. Across four benchmarks, SeKV improves over the strongest semantic compression baseline by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context. Code is available on https://github.com/AmirAbaskohi/SeKV.

Problem

• KV cache
• long-context
• memory bottleneck
• compression
• semantic preservation

Innovation

• resolution-adaptive KV cache
• hierarchical semantic memory
• low-rank SVD reconstruction
• on-demand token-level retrieval
• long-context LLM inference