Approaching Shannon Bound with Lossless LLM Weight Compression

2026-06-14

AI Summary

This work addresses the challenge of deploying trillion-byte-scale large language model (LLM) weights that far exceed GPU memory capacity, noting that existing systems underutilize their lossless compressibility. The study reveals, for the first time, that LLM weights exhibit substantial statistical redundancy and low effective entropy. To exploit this, the authors propose a block-wise just-in-time lossless decompression framework aligned with GPU GEMM tiling, built upon Asymmetric Numeral Systems (ANS). The framework supports diverse quantization formats—including bf16, int4, AWQ, and SQ8—and integrates seamlessly into the SGLang multi-GPU serving system. Without altering weight values, it approaches the Shannon compression limit, achieving significant throughput gains: batch size increases from 47 to 75 (1.2× throughput gain) on Qwen-14B and from 20 to 95 (4.8× larger batch, 1.6× throughput gain) on Mixtral-176B, outperforming NeuZip and DFloat11 by up to 11× in speed.

Abstract

Large language models (LLMs) now scale to trillions of parameters, driving weight storage into the terabyte regime and creating an acute mismatch with GPU memory capacity. Although lossless compression is widely effective in other domains, it remains underutilized in LLM systems. Through a comprehensive entropy study across models from 1.5B to 405B parameters and numeric formats ranging from bf16 to int4 and AWQ/SQ8, we find that LLM weights contain far less intrinsic randomness than their stored bitwidth implies, their effective entropy is 2-10x lower, indicating that up to a 10x footprint reduction is theoretically achievable without altering any weight values. Leveraging this insight, we introduce a tile-level, on-the-fly lossless decompression framework based on Asymmetric Numeral Systems that aligns decoding with the GEMM tiling pattern of GPU inference. Our design achieves bit-rates within 0.01-0.1 bits of the Shannon limit across a wide range of LLM numerical formats, demonstrating that nearly all statistical redundancy is eliminated. Integrated into the SGLang serving framework with multi-GPU support, our approach increases the maximum batch size of Qwen-14B from 47 to 75, improving throughput by up to 1.2x. On Mixtral-176B, the feasible batch size increases from 20 to 95 (4.8x), yielding up to 1.6x throughput improvement. Compared to state-of-the-art lossless compression approaches NeuZip and DFloat11, our design further improves throughput by up to 11x.

Problem

• large language models
• weight compression
• GPU memory
• lossless compression
• storage bottleneck

Innovation

• lossless compression
• Shannon limit
• Asymmetric Numeral Systems
• LLM weight entropy
• tile-level decompression