MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization

2026-06-14

AI Summary

This work addresses the challenge that 4-bit quantization struggles to simultaneously preserve accuracy for common values (inliers) and handle sparse, large-magnitude outliers, leading to significant degradation in large language model performance. Existing mixed-precision approaches compromise the uniformity of low-bit execution, limiting practical acceleration. To overcome this, the paper proposes MosaicQuant, the first unified 4-bit quantization framework that decouples inliers and outliers by decomposing weights into a dense 4-bit base component and a sparse 4-bit residual component, effectively modeling inliers while precisely compensating for outlier-induced errors. Furthermore, the authors introduce ZipperEngine, a unified GEMM inference pipeline that seamlessly integrates dense and sparse computation. Evaluated on LLaMA3 and Qwen3, MosaicQuant achieves near-FP16 accuracy while delivering up to 1.24× speedup over W16A16 baselines, striking an optimal balance between precision and efficiency.

Abstract

4-bit quantization significantly reduces the memory footprint and accelerates the inference of large language models (LLMs). However, its limited bit-width representation struggles to faithfully capture both dense common values (\emph{inliers}) and rare large-magnitude values (\emph{outliers}), causing substantial accuracy degradation. Existing mixed-precision methods mitigate this by retaining outliers in high precision, but at the cost of breaking the uniformity of low-bit execution, introducing precision conversion and extra data movement that undermine practical speedup. We propose \textbf{MosaicQuant}, a unified 4-bit LLM quantization paradigm built on a novel principle of \emph{inlier--outlier disaggregation}. Rather than elevating outlier precision, MosaicQuant quantizes the full weight matrix into a dense 4-bit base component, where inliers are captured faithfully while outlier are inevitably quantized. A sparse 4-bit residual component is then introduced to compensate for these quantization errors, selectively targeting the most error-critical weight blocks where output distortion is shown to be concentrated. However, a unified representation alone is insufficient, as naïvely executing the sparse residual as a separate kernel still breaks the unified low-bit inference pipeline. To bridge this gap, we introduce \textbf{ZipperEngine}, which fuses sparse block computation into the dense 4-bit GEMM kernel via an overlapped pipeline, unifying not only the representation but also the execution into a single coherent low-bit inference pipeline. Extensive experiments on LLaMA3 and Qwen3 demonstrate that MosaicQuant preserves near-FP16 accuracy while achieving up to $1.24\times$ speedup over the W16A16 baseline.

Problem

• 4-bit quantization
• inliers
• outliers
• large language models
• mixed-precision

Innovation

• inlier-outlier disaggregation
• unified 4-bit quantization
• sparse residual compensation
• low-bit GEMM fusion
• ZipperEngine