CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration

2026-08-31

AI Summary

为解决长上下文任务中大模型的内存需求和推理延迟问题，提出CateKV方法，通过识别并保留关键注意力头的信息减少KV缓存大小和计算开销。

Abstract

Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challenging due to the substantial memory requirements and inference latency. In this work, we discover that certain attention heads exhibit sequential consistency in their attention patterns, which can be persistently identified using a coefficient-of-variation-based algorithm. Inspired by this observation, we propose CateKV, a hybrid KV cache method that retains only critical token information for consistent heads, thereby reducing KV cache size and computational overhead, while preserving the majority of KV pairs in adaptive heads to ensure high accuracy. We show the unique characteristics of our algorithm and its extension with existing acceleration methods. Comprehensive evaluations on long-context benchmarks show that, while maintaining accuracy comparable to full attention, CateKV reduces memory usage by up to $2.72\times$ and accelerates decoding by $2.18\times$ in single-sample inputs, and boosts throughput by $3.96\times$ in batch scenarios.

Problem

• large language models
• long-context tasks
• memory requirements
• inference latency

Innovation

• Sequential Consistency
• Coefficient-of-variation-based Algorithm
• Hybrid KV Cache
• Long-context LLM Inference Acceleration
• Memory Reduction