KronQ: LLM Quantization via Kronecker-Factored Hessian

2026-07-08

AI Summary

This work addresses the performance degradation of existing second-order post-training quantization methods at low bit-widths, which stems from their neglect of the heterogeneous impact of output channels on reconstruction error. To overcome this limitation, the authors propose KronQ, a framework that leverages Kronecker decomposition to approximate the Hessian matrix, jointly models the covariance of activations and gradients, and applies bidirectional decorrelation across input and output dimensions to significantly enhance quantization accuracy. Furthermore, they introduce a cross-layer sensitivity metric based on the trace of the Hessian to enable automatic mixed-precision allocation. Evaluated on LLaMA-3-70B, KronQ achieves a WikiText-2 perplexity of 7.93 using only 2-bit weights, substantially outperforming baseline methods such as GPTQ.

Abstract

Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Existing second-order PTQ methods, including GPTQ, construct quantization objectives exclusively from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), KronQ achieves 7.93 perplexity.

Problem

• Post-training quantization
• Large Language Models
• Kronecker-factored Hessian
• Mixed-precision allocation
• Quantization sensitivity

Innovation

• Kronecker-factored Hessian
• post-training quantization
• gradient covariance
• mixed-precision allocation
• bidirectional incoherence