Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference

2025-02-02

AI Summary

To address memory and bandwidth bottlenecks hindering large language model (LLM) deployment on edge devices, this paper proposes the first end-to-end lossless compression framework tailored for LLMs, enabling compressed storage and direct inference across the full stack—cloud, disk, main memory, and on-chip caches. Methodologically, it integrates weight-distribution-adaptive Huffman coding, support for direct computation in the compressed domain, and memory- and bandwidth-aware weight repartitioning. Key contributions include: (i) strict preservation of original model behavior with zero precision loss; (ii) substantial reduction in weight loading bandwidth and on-chip storage footprint; (iii) improved inference latency and energy efficiency; and (iv) enabling efficient deployment of larger-scale LLMs on resource-constrained edge hardware. Experimental results demonstrate consistent accuracy retention while achieving up to 2.1× bandwidth savings and 1.8× on-chip memory reduction across diverse LLM architectures and edge platforms.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Data Compression

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

As they become more capable, large language models (LLMs) have continued to rapidly increase in size. This has exacerbated the difficulty in running state of the art LLMs on small, edge devices. Standard techniques advocate solving this problem through lossy compression techniques such as quantization or pruning. However, such compression techniques are lossy, and have been shown to change model behavior in unpredictable manners. We propose Huff-LLM, an emph{end-to-end, lossless} model compression method that lets users store LLM weights in compressed format emph{everywhere} -- cloud, disk, main memory, and even in on-chip memory/buffers. This allows us to not only load larger models in main memory, but also reduces bandwidth required to load weights on chip, and makes more efficient use of on-chip weight buffers. In addition to the memory savings achieved via compression, we also show latency and energy efficiency improvements when performing inference with the compressed model.

Problem

• Large Language Models
• Resource-efficient Deployment
• Information Preservation

Innovation

• Huff-LLM
• Lossless Compression
• Large Language Models