70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float

2025-04-15

AI Summary

To address storage and inference efficiency bottlenecks in deploying large language models (LLMs) on resource-constrained GPUs, this paper proposes DFloat11—the first lossless, dynamic-length floating-point compression framework ensuring bit-identical output consistency. Its core contributions are: (1) information-theoretically optimal, dynamic-length entropy coding tailored to LLM weight distributions; (2) an SRAM-efficient, block-wise lookup table (LUT) design coupled with a two-stage, thread-cooperative GPU decompression kernel; and (3) a transformer-block-level online decompression scheduling strategy. Evaluated on Llama-3.1, Qwen-2.5, and Gemma-3, DFloat11 achieves 30% model size reduction with full bit-identical accuracy. It improves inference throughput by 1.9×–38.8× and extends context length by 5.3×–13.17× under identical GPU memory constraints. Notably, it enables deployment of the 810GB Llama-3.1-405B model on a single node with eight 80GB GPUs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) have grown rapidly in size, creating significant challenges for efficient deployment on resource-constrained hardware. In this paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression framework that reduces LLM size by 30% while preserving outputs that are bit-for-bit identical to the original model. DFloat11 is motivated by the low entropy in the BFloat16 weight representation of LLMs, which reveals significant inefficiency in existing storage format. By applying entropy coding, DFloat11 assigns dynamic-length encodings to weights based on frequency, achieving near information-optimal compression without any loss of precision. To facilitate efficient inference with dynamic-length encodings, we develop a custom GPU kernel for fast online decompression. Our design incorporates the following: (i) decomposition of memory-intensive lookup tables (LUTs) into compact LUTs that fit in GPU SRAM, (ii) a two-phase kernel for coordinating thread read/write positions using lightweight auxiliary variables, and (iii) transformer-block-level decompression to minimize latency. Experiments on recent models, including Llama-3.1, Qwen-2.5, and Gemma-3, validates our hypothesis that DFloat11 achieves around 30% model size reduction while preserving bit-for-bit exact outputs. Compared to a potential alternative of offloading parts of an uncompressed model to the CPU to meet memory constraints, DFloat11 achieves 1.9-38.8x higher throughput in token generation. With a fixed GPU memory budget, DFloat11 enables 5.3-13.17x longer context lengths than uncompressed models. Notably, our method enables lossless inference of Llama-3.1-405B, an 810GB model, on a single node equipped with 8x80GB GPUs. Our code and models are available at https://github.com/LeanModels/DFloat11.

Problem

• Reduces LLM size by 30% without losing accuracy
• Improves GPU inference efficiency via dynamic-length encoding
• Enables lossless compression for large models like Llama-3.1-405B

Innovation

• Dynamic-Length Float (DFloat11) for lossless compression
• Entropy coding for near-optimal weight compression
• Custom GPU kernel for fast online decompression