Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression

2026-09-02

AI Summary

本文提出了一种基于Kronecker的可扩展近似方法，用于分析十亿参数模型的Hessian矩阵，无需存储整个Fisher矩阵，从而识别脆弱组件以指导压缩和优化策略。

Abstract

In this paper, we propose a scalable Kronecker-based approximation that captures cross-layer interactions without storing the entire Fisher matrix, enabling practical Hessian analysis for billion-parameter networks where full computation is infeasible. Our approach reveals consistent vulnerability patterns: value projection layers exhibit the highest sensitivity and strongest cross-layer correlations across multiple model families, while other components exhibit architecture-specific behaviors. Through extensive experiments on quantization, sparsification, inter-layer corruption, and post-corruption fine-tuning, we demonstrate that our approximation strongly correlates with both performance degradation and recovery. Our framework provides a practical, theoretically grounded tool for identifying fragile components in large models, opening new avenues for guided compression and optimization strategies, such as mixed-precision allocation, layer-wise sparsity, and adaptive low-rank decomposition across layers and even individual weight groups.

Problem

• Hessian analysis
• billion-parameter networks
• Fisher matrix
• cross-layer interactions
• compression

Innovation

• Scalable Kronecker-based approximation
• Cross-layer interactions
• Billion-parameter networks
• Hessian analysis
• Model compression