RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

2026-06-30

AI Summary

This work addresses the substantial memory overhead of KV caching in long-context large language model inference, a challenge inadequately tackled by existing sparse attention methods that either rely on static budgets or suffer from biased proxy scores computed at high computational cost. The authors propose RaBitQCache, a framework featuring an unbiased proxy scoring mechanism with theoretically bounded approximation error, enabling dynamic Top-p token selection. It efficiently estimates attention weights through randomized rotation-based binary quantization and binary-INT4 arithmetic, while integrating asynchronous pipelining and lazy-update strategies to minimize system-level overhead. Extensive experiments demonstrate that RaBitQCache significantly reduces memory I/O and inference latency while achieving superior generation quality compared to state-of-the-art baselines.

Abstract

Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased. To address these limitations, we propose RaBitQCache, a novel sparse attention framework that utilizes randomized rotated binary quantization and high-throughput binary-INT4 arithmetic to efficiently estimate attention weights. Our proxy score serves as an unbiased estimator with a proven error bound, enabling adaptive Top-p retrieval that dynamically adjusts the token budget based on actual attention sparsity. We further implement a hardware-aware system with asynchronous pipelining and lazy updates to mask overhead. Evaluations demonstrate that RaBitQCache significantly accelerates inference and reduces memory I/O while preserving generation quality compared to state-of-the-art baselines. Code is available at https://github.com/Sakuraaa0/RaBitQCache.git.

Problem

• KVCache
• long-context LLM inference
• sparse attention
• attention sparsity
• memory bottleneck

Innovation

• rotated binary quantization
• sparse attention
• adaptive Top-p retrieval
• KV cache compression
• binary-INT4 arithmetic