Trainable Smooth-Rotation Transforms with Learned Channel Scales for LLM Quantization

📅 2026-06-07
📈 Citations: 0
Influential: 0
📄 PDF
🤖 AI Summary
This work addresses the issue of activation quantization error in post-training quantization of large language models, where outlier-dominated channels lead to excessive weight migration under conventional max-based scaling strategies. To mitigate this, the authors propose a joint optimization approach that replaces maximum-value statistics with robust high-percentile scaling and learns channel-wise scaling factors through constrained gradient-based optimization, all within the SmoothQuant-equivalent transformation framework. Experiments on LLaMA-3.2-1B demonstrate that the method reduces quantization error by 18.5% in selected layers and lowers the average error across all layers from 97.51 to 78.08—a 19.9% improvement—significantly enhancing the accuracy of W4A4 post-training quantization.
📝 Abstract
Post-training quantization (PTQ) is one of the most practical ways to reduce the serving cost of Large Language Models (LLMs), but activation quantization remains difficult because outlier-dominated channels lead to large quantization errors. This paper investigates whether part of this degradation is caused by over-migration in scaling-based equivalent transformations. We introduce a quantile-robust scaling policy for SmoothRot-style transforms by replacing max-based activation statistics with high quantiles, and we complement it with constrained gradient-based optimization of channel scales. On LLaMA-3.2-1B under W4A4 quantization, quantile-only policy search improves selected-layer error by 11.1% over the SmoothRot baseline, joint (alpha, q) search improves it by 12%, and training reaches 18.5%. Replaying the best selected-layer policy on all decoder-block down-projection layers reduces the corresponding full-layer mean error from 97.51 to 78.08 (19.9%). The results show that robust migration control and lightweight scale learning provide consistent gains over max-based fixed policies while preserving the equivalent-transform framework.
Problem

Research questions and friction points this paper is trying to address.

post-training quantization
activation quantization
outlier channels
quantization error
equivalent transformations
Innovation

Methods, ideas, or system contributions that make the work stand out.

quantile-robust scaling
SmoothRot
channel scale learning
post-training quantization
outlier mitigation
💼 Related Jobs
No related jobs found.
P
Patrik Czakó
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University, Budapest, Hungary
G
Gábor Kertész
John von Neumann Faculty of Informatics, Obuda University, Budapest, Hungary
S
Sándor Szénási
John von Neumann Faculty of Informatics, Obuda University, Budapest, Hungary