SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

2026-05-03

AI Summary

This work addresses the performance bottleneck caused by cross-node transmission of KV caches in disaggregated prefill-decode architectures for large language model services. We propose the first GPU-accelerated online lossless compression scheme tailored for KV cache tensors. Our method exploits redundancy in the BF16 floating-point exponents of KV activations by combining an offline-calibrated exponent codebook with fixed-length encoding for frequent exponents and a sparse escape stream for rare ones, forming a dense-sparse hybrid encoding pipeline. With GPU-optimized parallel encoding and decoding, our approach achieves 613.3 GB/s compression and 2181.8 GB/s decompression throughput on real KV tensors, yielding a 1.32× end-to-end transmission speedup, a 1.30× reduction in time-to-first-token latency, and a 1.23× improvement in request throughput.

Abstract

Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to better load-balance between the compute-bound prefill phase and the memory-bound decode phase. Under this design, prefill workers generate a KV cache that must be transferred to decode workers before token generation can begin. With these workers residing on different physical systems, this transfer becomes a significant bottleneck to serving LLMs at scale. This bottleneck gets exacerbated for long-input and agentic workloads, which typically require long inputs. Existing lossless codecs are not well suited to this setting as they primarily target offline weight compression, rely on CPU-side, or use variable-length coding that decompresses fast but compresses too slowly for online use. SplitZip is a GPU-friendly lossless compressor for KV-cache transfer. It exploits redundancy in floating-point exponents of KV activations, encoding the most frequent exponent values with fixed-length codes, and encoding (position, value) pairs and value of rare exponents in an escape stream. An offline calibrated top-16 exponent codebook enables online encoding, while the regular dense path and sparse escape correction make both encoding and decoding efficient on GPUs. On real BF16 activation tensors, SplitZip achieves 613.3 GB/s compression throughput and 2181.8 GB/s decompression throughput, substantially outperforming prior lossless compressors on the latency-critical codec path. End-to-end transfer experiments show up to 1.32$\times$ speedup for BF16 KV-cache transfer, 1.30$\times$ speedup for TTFT and 1.23$\times$ increase on Request Throughput.

Problem

• KV cache compression
• disaggregated LLM serving
• lossless compression
• prefill-decode disaggregation
• online compression

Innovation

• KV cache compression
• lossless compression
• GPU acceleration
• prefill-decode disaggregation
• exponent redundancy