Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation

2026-05-28

AI Summary

KV cache management is a major bottleneck in long-context generation for large language models. Existing compression approaches often degrade critical context during the prefill phase, while strategies tailored to the decoding phase remain underexplored. This work proposes a novel decoding-phase-only KV cache compression method that introduces, for the first time, a momentum-driven temporal attention aggregation mechanism. This mechanism dynamically models token importance by jointly accounting for long-term influence and recent relevance. Through momentum-based attention decay aggregation, the method achieves dynamic KV cache compression without increasing decoding latency, significantly improving generation quality by 2.3–3.2% over existing approaches. It thereby overcomes the limitations of static heuristics or instantaneous attention-based methods.

Abstract

Key-Value (KV) cache remains a major bottleneck for deploying Large Language Models (LLMs) in long-generation tasks. Prior work often applies uniform compression across both prefill and decoding caches, but compressing the prefill cache degrades performance by corrupting critical context. While preserving the prefill cache is essential, decoding-phase compression remains underexplored, with existing methods relying on rigid recency windows or instantaneous attention. Our analysis of attention dynamics reveals strong temporal patterns: critical tokens receive sustained attention over long horizons, while local reasoning involves short-lived bursts. Static heuristics fail to capture this behavior, leading to premature eviction of important tokens or retention of stale ones. We propose Moment-KV, a decoding-time KV cache compression method based on momentum-driven temporal attention aggregation. Our method models token importance as a continuously evolving state, where attention is aggregated with decay, capturing both long-term influence and recent relevance. Experiments show that Moment-KV significantly improves generation fidelity in long-generation tasks (2.3-3.2 %) while maintaining decoding latency.

Problem

• KV cache compression
• long-generation
• attention dynamics
• decoding-phase
• memory bottleneck

Innovation

• KV cache compression
• momentum-based attention
• decode-time optimization
• temporal attention dynamics
• long-generation LLMs