SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression

2024-10-12

AI Summary

To address the insufficient accuracy of one-shot compression methods and the difficulty of synergistically integrating multiple compression paradigms for large language models (LLMs), this paper proposes the first fine-tuning-free one-shot weight compression framework. It unifies hardware-friendly 4-bit stochastic uniform quantization (SLIM-Quant), 2:4 semi-structured pruning, and low-rank compensation guided by a reversible saliency function. Theoretical analysis derives closed-form low-rank adapter parameters to ensure mathematically tractable compensation for quantization and pruning errors. Evaluated on LLaMA-2-7B under 2:4 sparsity plus 4-bit quantization, the method achieves a 5.66% accuracy improvement over baseline one-shot methods; inference latency is reduced by 3.78× on RTX 3060 and 3.75× on A100. With optional PEFT fine-tuning, accuracy further improves by 1.66%. This work establishes a principled, efficient, and modular foundation for high-fidelity, hardware-efficient LLM compression.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Data Compression

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Conventional model compression techniques for LLMs address high memory consumption and slow inference challenges but typically require computationally expensive retraining to preserve accuracy. In contrast, one-shot compression methods eliminate retraining cost, but struggle to achieve accuracy comparable to dense models. This paper presents SLIM, a new one-shot compression framework that holistically integrates hardware-friendly quantization, sparsity, and low-rank approximation into a unified process. First, we formulate the quantization process using a probabilistic approach (SLIM-Quant) that enables us to apply uniform quantization. Then, we use an existing one-shot pruning method to apply semi-structured sparsity on top of the quantized weights. Finally, to compensate for the introduced aggregated quantization and sparsity error, we use a novel saliency function with unique invertible and additive features that enables us to mathematically compute the value of low-rank adapters. SLIM improves model accuracy by up to 5.66% (LLaMA-2-7B) for 2:4 sparsity with 4-bit weight quantization, outperforming prior methods. Models compressed with SLIM achieve up to 3.78x and 3.75x layer-wise speedup on Nvidia RTX3060 and A100 GPUs, respectively. We also propose an optional PEFT recipe that further improves accuracy by up to 1.66% (LLaMA-2-13B) compared to SLIM without fine-tuning

Problem

• Reduces LLM memory consumption and speeds inference.
• Integrates quantization, sparsity, and low-rank approximation.
• Improves accuracy without costly retraining.

Innovation

• One-shot compression integrates quantization, sparsity, low-rank.
• Probabilistic approach enables uniform weight quantization.
• Novel saliency function computes low-rank adapter values.