EpiCache: Episodic KV Cache Management for Long Conversational Question Answering

2025-09-22

AI Summary

To address the linear growth of KV cache memory with context length in long conversations—and the limitations of existing compression methods, which either cause uncontrolled peak memory usage or loss of contextual coherence—this paper proposes Episodic KV Compression. Our method organizes dialogue history into coherent “episodic segments” via training-free chunked prefilling and clustering-driven semantic segmentation. Leveraging layer-wise sensitivity analysis and dynamic memory budget allocation, it enables segment-level, query-agnostic cache eviction. Crucially, this avoids both the peak memory surge from full-context prefilling and the degradation of multi-turn coherence caused by query-dependent eviction strategies. Evaluated on three long-context question-answering benchmarks, our approach achieves up to 40% accuracy improvement, maintains near-full-cache performance at 4–6× compression ratios, and reduces inference latency and memory footprint by up to 2.4× and 3.5×, respectively.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent advances in large language models (LLMs) have extended context lengths, enabling assistants to sustain long histories for coherent, personalized responses. This ability, however, hinges on Key-Value (KV) caching, whose memory grows linearly with dialogue length and quickly dominates under strict resource constraints. An active line of research for reducing this overhead is KV cache compression, which seeks to limit cache size while preserving accuracy. Yet existing methods face two major limitations: (i) evicting entries after full-context prefill causes unbounded peak memory, and (ii) query-dependent eviction narrows the cache to a single query, leading to degraded accuracy in multi-turn conversations. We introduce EpiCache, a training-free KV cache management framework for long conversational question answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression, which clusters conversation history into coherent episodes and applies episode-specific KV cache eviction. We further design an adaptive layer-wise budget allocation strategy that measures each layer's sensitivity to eviction and distributes the memory budget across layers accordingly. Across three LongConvQA benchmarks, EpiCache improves accuracy by up to 40% over recent baselines, sustains near-full KV accuracy under 4-6x compression, and reduces latency and memory by up to 2.4x and 3.5x, thereby enabling efficient multi-turn interaction under strict resource constraints.

Problem

• Managing KV cache memory growth during long conversations under resource constraints
• Preventing accuracy degradation when compressing KV cache for multi-turn dialogues
• Optimizing memory allocation across layers while maintaining conversational coherence

Innovation

• Block-wise prefill for bounded cache growth
• Episodic KV compression via topic clustering
• Adaptive layer-wise budget allocation strategy