Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

2026-07-02

AI Summary

This work addresses the high decoding latency in long-context inference caused by full transmission of KV cache, a challenge exacerbated by the difficulty of existing quantization methods to simultaneously achieve low latency and high accuracy. The authors propose a progressive speculative quantization mechanism that partitions the KV cache into a high-priority Anchor stream and a low-priority Residual stream based on bit significance, thereby relaxing the assumption that the entire cache must be received before decoding can begin. This enables decoding to commence as soon as critical bits arrive, while residual data is asynchronously loaded in the background. By integrating dual-stream parallel transmission, speculative decoding, and result verification, the method achieves first-token latency (TTFT) comparable to 4-bit quantization across diverse models and workloads, while preserving BF16-level accuracy—yielding up to a 1.43× speedup and a 5.1% accuracy improvement over 8-bit quantization.

Abstract

Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems. In disaggregated inference, these workloads require transferring large Key-Value (KV) caches across the network, where decoding cannot begin until the transfer completes. Recent KV quantization techniques reduce data volume and alleviate this bottleneck, but existing schemes fail to achieve both low network-exposed latency and high inference accuracy. We challenge the assumption that the KV cache is an indivisible unit that must be fully received before use. We leverage the observation that different bits in the KV cache contribute unequally to attention computation and inference precision: the most significant bits capture the coarse structure of attention and the least significant bits refine precision. This property enables partial use of the KV cache during decoding. We present Lynx, a system that enables progressive, split-stream KV transfer by partitioning the KV cache into a high-priority Anchor stream carrying the most significant bits and a low-priority Residual stream carrying remaining precision. Decoding begins upon receipt of the Anchor stream and proceeds speculatively while the Residual stream is transferred concurrently, followed by verification that ensures equivalence to higher-precision decoding. Across multiple models and serving workloads, Lynx achieves Time-to-First-Token (TTFT) comparable to aggressive 4-bit KV quantization, while matching the accuracy of high-precision (BF16) inference, improving TTFT over standard 8-bit KV quantization by up to $1.43\times$ and improving accuracy over state-of-the-art by up to $5.1\%$.

Problem

• long-context inference
• KV cache transfer
• network latency
• inference accuracy
• KV quantization

Innovation

• progressive quantization
• speculative decoding
• KV cache transfer
• long-context inference
• disaggregated serving