Problem
Research questions and friction points this paper is trying to address.
Kronecker-factored Hessian
post-training quantization
Hessian approximation
WaterKron and FlipFlop Hessian: Information-Theoretically Grounded Quantization with Kronecker-factored Hessians
🤖 AI Summary
研究通过WaterKron和FlipFlop Hessian方法解决了如何选择克罗内克分解的海森矩阵近似用于后训练量化的问题，利用信息论基础优化了量化过程。
📝 Abstract
How should a Kronecker-factored Hessian approximation be chosen for post-training quantization? We address this question through WaterKron, which combines two-sided GPTQ with row- and column-dependent waterfilling scales and entropy coding. We derive its high-rate distortion with respect to the full Hessian using an explicit Kronecker-Hessian mismatch factor $Φ$. This factor quantifies the asymptotic distortion penalty due to the Kronecker Hessian approximation and provides a criterion for selecting the factors optimally. Minimizing $Φ$ leads to a Gaussian covariance-fitting problem with classical ``flip-flop'' updates. We thus give a rate-distortion justification for using the resulting FlipFlop Hessian in quantization. We evaluate it empirically, finding that the FlipFlop Hessian consistently improves KL divergence and perplexity over input-only, marginal, and Frobenius-based Hessian choices.
Innovation
Methods, ideas, or system contributions that make the work stand out.
WaterKron
FlipFlop Hessian
Kronecker-factored Hessian
entropy coding
rate-distortion
🔎 Similar Papers
No similar papers found.
