CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference

2026-02-24

AI Summary

This work addresses the high latency in long-context large language model inference caused by the growing key-value (KV) cache, a challenge exacerbated by existing pruning methods that lack contextual awareness and struggle to balance generation quality with acceleration. To overcome this, the authors propose CHESS, a co-designed algorithmic and systems-level KV cache management framework that introduces, for the first time, a context-aware hierarchical semantic selection mechanism. This mechanism dynamically reconstructs the critical context required for decoding while leveraging coarse-grained cache operations to minimize data movement overhead. Experiments demonstrate that CHESS achieves superior generation quality compared to full-cache baselines while retaining only 1% of the KV cache, and it improves inference throughput by up to 4.56×, significantly outperforming current state-of-the-art approaches.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Long-context LLMs demand accurate inference at low latency, yet decoding becomes primarily constrained by KV cache as context grows. Prior pruning methods are largely context-agnostic: their token selection ignores step-wise relevance and local semantics, which undermines quality. Moreover, their irregular accesses and selection overheads yield only limited wall-clock speedups. To address this, we propose \textbf{CHESS}, an \textit{algorithm-system co-design} KV-cache management system. Algorithmically, CHESS introduces a context-aware, hierarchical selection policy that dynamically reconstructs a coherent context for the current decoding. System-wise, coarse granularity selection eliminates expensive data movement, fully realizing practical acceleration from theoretical sparsity. Extensive evaluations demonstrate that CHESS surpasses Full-KV quality using only \textbf{1\%} of the KV cache, delivers low-latency stable inference with up to \textbf{4.56$\times$} higher throughput, and consistently outperforms other strong baselines. Code is available at \href{https://anonymous.4open.science/r/CHESS-9958/}{https://anonymous.4open.science/r/CHESS/}.

Problem

• long-context LLM
• KV cache
• context-aware pruning
• inference latency
• semantic selection

Innovation

• context-aware
• hierarchical selection
• KV cache pruning
• algorithm-system co-design
• long-context LLM