OSC: Hardware Efficient W4A4 Quantization via Outlier Separation in Channel Dimension

2026-04-14

AI Summary

This work addresses the significant accuracy degradation in large language models under 4-bit quantization, primarily caused by activation outliers. The study reveals, for the first time, that these outliers exhibit stable, structured clustering across tokens along the channel dimension. Building on this insight, the authors propose the Outlier-Structured Clustering (OSC) framework, which identifies outlier channels offline and dynamically extracts structured sub-tensors online to enable a dual-path GEMM: one path operates at low precision (W4A4), while the other retains high precision (W16A16). An FP8 fallback mechanism further enhances robustness. The design is hardware-friendly and seamlessly integrates with existing 4-bit microarchitectures. Evaluated on Qwen3-8B and Qwen3-30B, OSC incurs only 2.19 and 1.12 average accuracy points of loss, respectively, and achieves up to 1.78× speedup over W8A8 GEMM on modern AI accelerators.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

While 4-bit quantization is essential for high-throughput deployment of Large Language Models, activation outliers often lead to significant accuracy degradation due to the restricted dynamic range of low-bit formats. In this paper, we systematically investigate the spatial distribution of outliers and demonstrate a token-persistent structural clustering effect, where high-magnitude outliers consistently occupy fixed channels across tokens. Building on this insight, we propose OSC, a hardware-efficient framework for outlier suppression. During inference, OSC executes a dual-path computation consisting of a low-precision 4-bit General Matrix Multiplication (GEMM) path and a high-precision 16-bit branch GEMM path. Specifically, OSC uses an offline group-wise strategy to identify the channels where outliers are located and then performs structured sub-tensor extraction to coalesce these scattered activation channels into a compact dense tensor online. This mechanism implements outlier protection through regularized and high-throughput GEMM operations, achieving a seamless fit with modern 4-bit micro-scaling hardware. Furthermore, for the inputs of W2 where outlier clustering is less pronounced, we integrate a fallback strategy to FP8. Evaluation on Qwen3-8B and Qwen3-30B restricts the average accuracy drop to 2.19 and 1.12 points, respectively. Notably, OSC is highly hardware-friendly, achieving a peak speedup of 1.78x over the W8A8 GEMM baseline on a modern AI accelerator.

Problem

• 4-bit quantization
• activation outliers
• Large Language Models
• accuracy degradation
• dynamic range

Innovation

• outlier separation
• W4A4 quantization
• structured sparsity
• dual-path GEMM
• hardware-efficient LLM inference