CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection

2026-05-16

AI Summary

This work addresses the challenge of balancing computational efficiency and key-value (KV) cache access accuracy in existing sparse attention methods under chunked prefill scenarios. The authors propose CompactAttention, a novel mechanism that reformulates two-dimensional block-sparse masks into group-query attention (GQA)-aware per-group KV block tables, which serve as KV selection signals rather than explicit sparse kernel execution plans. By leveraging joint query-block and intra-group coalescing strategies, CompactAttention constructs minimal block tables to enable in-place, efficient KV access—eliminating explicit compression overhead while remaining compatible with existing paged attention systems. Evaluated on LLaMA-3.1-8B-Instruct with a 128K context length, the method achieves up to a 2.72× speedup in attention computation while preserving accuracy nearly on par with dense attention on the RULER benchmark.

Abstract

Chunked prefill has become a widely adopted serving strategy for long-context large language models, but efficient attention computation in this regime remains challenging. Existing sparse attention methods are primarily designed for one-shot prefill and do not translate efficiently to chunked prefill: block-sparse kernels lose efficiency when the query length is limited by the chunk size, while fine-grained pattern search becomes costly when repeated over the accumulated KV cache at every chunk. QUOKA, a recent method that directly targets chunked prefill, avoids sparse-kernel overhead but relies on query-subsampled, token-level KV selection, which can miss query-specific KV entries and introduce explicit KV-copy overhead. To address these limitations, we propose CompactAttention, a chunked-prefill attention mechanism based on Block-Union KV Selection. CompactAttention treats 2D block-sparse masks as KV-selection signals rather than direct sparse-kernel execution plans, and converts them into GQA-aware per-group KV block tables through Q-block union and intra-group union. This construction produces the minimal block tables that preserve all KV blocks selected by the input masks under paged execution constraints, enabling selected KV blocks to be accessed in place without explicit KV compaction. On LLaMA-3.1-8B-Instruct, CompactAttention maintains accuracy close to dense attention on the RULER benchmark while delivering up to 2.72$\times$ attention speedup at 128K context length under chunked prefill.

Problem

• chunked prefill
• attention computation
• KV selection
• sparse attention
• long-context LLMs

Innovation

• Chunked Prefill
• Block-Union KV Selection
• Sparse Attention
• KV Cache Optimization
• GQA-aware Block Tables