FlatQuant: Flatness Matters for LLM Quantization

2024-10-12

arXiv.org

AI Summary

To address the substantial accuracy degradation in post-training quantization of large language models (LLMs) caused by weight and activation outliers, this paper proposes a lightweight learnable affine flattening method. Specifically, it designs layer-specific affine transformations for each linear module to improve numerical distribution uniformity and thereby reduce uniform quantization error. To ensure efficiency, the method employs Kronecker decomposition to compress transformation parameters and integrates full-operator fusion for single-kernel deployment. Evaluated on LLaMA-3-70B, the approach achieves W4A4 quantization with only 1% accuracy loss—outperforming SpinQuant by 7.5 percentage points. Moreover, it accelerates prefill and decoding latency by 2.3× and 1.7×, respectively, while incurring merely 0.07% additional runtime overhead. The method thus delivers a practical, high-fidelity, and hardware-efficient solution for LLM quantization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recently, quantization has been widely used for the compression and acceleration of large language models~(LLMs). Due to the outliers in LLMs, it is crucial to flatten weights and activations to minimize quantization error with the equally spaced quantization points. Prior research explores various pre-quantization transformations to suppress outliers, such as per-channel scaling and Hadamard transformation. However, we observe that these transformed weights and activations can still remain steep and outspread. In this paper, we propose FlatQuant (Fast and Learnable Affine Transformation), a new post-training quantization approach to enhance flatness of weights and activations. Our approach identifies optimal affine transformations tailored to each linear layer, calibrated in hours via a lightweight objective. To reduce runtime overhead, we apply Kronecker decomposition to the transformation matrices, and fuse all operations in FlatQuant into a single kernel. Extensive experiments show that FlatQuant sets up a new state-of-the-art quantization benchmark. For instance, it achieves less than $ extbf{1}%$ accuracy drop for W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by $ extbf{7.5}%$. For inference latency, FlatQuant reduces the slowdown induced by pre-quantization transformation from 0.26x of QuaRot to merely $ extbf{0.07x}$, bringing up to $ extbf{2.3x}$ speedup for prefill and $ extbf{1.7x}$ speedup for decoding, respectively. Code is available at: url{https://github.com/ruikangliu/FlatQuant}.

Problem

• Large Language Models
• Quantization
• Performance Degradation

Innovation

• FlatQuant
• Adaptive Transformation
• Matrix Decomposition