KVBuffer: IO-aware Serving for Linear Attention

2026-05-18

AI Summary

Existing linear attention services suffer from high memory overhead and inefficient inference due to frequent updates of large attention states at each decoding step. This work proposes KVBuffer, a novel mechanism that introduces the first I/O-aware buffering strategy for attention computation. By caching recent key-value pairs, KVBuffer enables chunked computation, batched state updates, parallel draft token verification, and stateless short-context attention. This approach substantially reduces memory traffic and latency, achieving up to a 45.17% reduction in decoding latency on Qwen3-Next and a fivefold increase in request throughput under speculative decoding with four draft tokens.

Abstract

Linear attention has recently gained significant attention for long-context inference due to its constant decoding cost with respect to context length. However, existing serving systems typically serve linear attention by recurrently computing and updating a large linear attention state in every decoding step. Since the state is much larger than the per-token key and value, recurrent decoding incurs substantial memory access and becomes inefficient for serving linear attention. In this paper, we propose KVBuffer, an IO-aware serving mechanism for linear attention. By buffering recent keys and values, KVBuffer enables serving systems to compute linear attention outputs in more flexible and memory-efficient ways. For decoding, KVBuffer enables chunkwise computation, which reduces average memory access and decoding latency by deferring state updates and applying them in batch. For speculative decoding, KVBuffer verifies draft tokens in parallel and avoids storing temporary states. For short contexts, KVBuffer computes attention outputs directly from buffered keys and values, without creating or updating the linear attention state. We implement KVBuffer in SGLang for Qwen3-Next. Our evaluations show that KVBuffer can reduce linear attention decoding latency by up to 45.17% and increase the maximum number of serving requests by 5x for speculative decoding when verifying four draft tokens.

Problem

• linear attention
• serving
• memory access
• decoding latency
• IO efficiency

Innovation

• linear attention
• KVBuffer
• IO-aware serving
• chunkwise decoding
• speculative decoding