🤖 AI Summary
This work addresses the issue of activation quantization error in post-training quantization of large language models, where outlier-dominated channels lead to excessive weight migration under conventional max-based scaling strategies. To mitigate this, the authors propose a joint optimization approach that replaces maximum-value statistics with robust high-percentile scaling and learns channel-wise scaling factors through constrained gradient-based optimization, all within the SmoothQuant-equivalent transformation framework. Experiments on LLaMA-3.2-1B demonstrate that the method reduces quantization error by 18.5% in selected layers and lowers the average error across all layers from 97.51 to 78.08—a 19.9% improvement—significantly enhancing the accuracy of W4A4 post-training quantization.
📝 Abstract
Post-training quantization (PTQ) is one of the most practical ways to reduce the serving cost of Large Language Models (LLMs), but activation quantization remains difficult because outlier-dominated channels lead to large quantization errors. This paper investigates whether part of this degradation is caused by over-migration in scaling-based equivalent transformations. We introduce a quantile-robust scaling policy for SmoothRot-style transforms by replacing max-based activation statistics with high quantiles, and we complement it with constrained gradient-based optimization of channel scales. On LLaMA-3.2-1B under W4A4 quantization, quantile-only policy search improves selected-layer error by 11.1% over the SmoothRot baseline, joint (alpha, q) search improves it by 12%, and training reaches 18.5%. Replaying the best selected-layer policy on all decoder-block down-projection layers reduces the corresponding full-layer mean error from 97.51 to 78.08 (19.9%). The results show that robust migration control and lightweight scale learning provide consistent gains over max-based fixed policies while preserving the equivalent-transform framework.