Technical Report: Activation Residual Hessian Quantization (ARHQ) for Low-Bit LLM Quantization

2026-04-30

AI Summary

This work addresses the severe error propagation in low-bit activation-weight quantization, which significantly degrades inference performance of large language models. The authors propose a post-training weight splitting method that constructs an input-side residual Hessian matrix using activation quantization residuals to analytically identify weight directions most sensitive to quantization error. These critical components are then precisely isolated into a high-precision, low-rank branch via closed-form truncated singular value decomposition, without requiring additional training. Innovatively leveraging residual Hessian estimation to guide low-rank compensation, the approach substantially improves inter-layer signal-to-noise ratio in the Qwen3-4B-Thinking-2507 model and maintains strong downstream performance on the ZebraLogic benchmark, demonstrating robustness even under extremely low-bit settings.

Abstract

We present Activation Residual Hessian Quantization (ARHQ), a post-training weight splitting method designed to mitigate error propagation in low-bit activation-weight quantization. By constructing an input-side residual Hessian from activation quantization residuals (G_x), ARHQ analytically identifies and isolates error-sensitive weight directions into a high-precision low-rank branch. This is achieved via a closed-form truncated SVD on the scaled weight matrix W G^{1/2}_x . Experimental results on Qwen3-4B-Thinking-2507 demonstrate that ARHQ significantly improves layer-wise SNR and preserves downstream reasoning performance on ZebraLogic even under aggressive quantization. The code is available at https://github.com/BeautMoonQ/ARHQ.

Problem

• low-bit quantization
• error propagation
• large language models
• activation quantization
• weight quantization

Innovation

• Activation Residual Hessian Quantization
• low-bit quantization
• error propagation mitigation
• truncated SVD
• post-training quantization