AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference

2024-11-15

arXiv.org

AI Summary

To address accuracy degradation in 4-bit LLM inference caused by activation outliers, this paper proposes AMXFP4—a calibration-free, asymmetric micro-scaled floating-point format. Departing from conventional symmetric MX constraints, AMXFP4 is the first to uncover a fundamental trade-off between intra-group asymmetry and outlier suppression. It employs group-level dynamic shared scaling to enable heterogeneous rescaling, jointly suppressing outliers while preserving representational fidelity. The method supports fully quantized matrix multiplication, incurs zero calibration overhead, and requires negligible additional hardware cost. Experiments demonstrate that AMXFP4 achieves a 3% improvement over MXFP4 on VQA, outperforms rotation-based methods by 1.6% on CSQA, and surpasses deployed commercial MXFP4 variants. Notably, it enables the first end-to-end 4-bit LLM inference with competitive accuracy.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

As large language models (LLMs) grow in parameter size and context length, computation precision has been reduced from 16-bit to 4-bit to improve inference efficiency. However, this reduction causes accuracy degradation due to activation outliers. Rotation-based INT4 methods address this via matrix calibration, but they introduce multi-hour overheads and leave key computations in full precision. Microscaling (MX) floating-point (FP) formats offer fine-grained representation with a shared scale, enabling fully quantized matrix multiplications through direct casting without calibration. However, existing research shows unsatisfactory empirical results for MXFP4 inference, and the robustness of MX formats remains largely unexplored. In this work, we uncover the fundamental tradeoffs of the MX format: while it effectively suppresses activation outliers, it does so at the cost of increased group-wise asymmetry. To address this, we propose AMXFP4, a 4-bit asymmetric FP format that handles both issues using asymmetric shared scales, without requiring calibration. Our custom MAC engine adds negligible hardware cost while improving accuracy: AMXFP4 outperforms MXFP4 by 3% on VQA and exceeds rotation-based methods by 1.6% on CSQA. It also surpasses recently deployed commercial MXFP4 variants. Code: https://github.com/aiha-lab/MX-QLLM

Problem

• Reducing 4-bit LLM inference accuracy loss from activation outliers
• Eliminating multi-hour calibration overheads in rotation-based INT4 methods
• Improving robustness of MXFP4 formats for fully quantized matrix multiplications

Innovation

• Asymmetric FP format for 4-bit precision
• Shared scales without calibration overhead
• Custom MAC engine for improved accuracy