Problem
Research questions and friction points this paper is trying to address.
model compression
functional degeneracy
behavioral recovery rank
Functional Degeneracy in Neural Networks: Measurement and Pruning
🤖 AI Summary
研究通过定义行为恢复秩来量化神经网络的功能退化，以此作为压缩模型的基准，发现结构和幅度修剪后仍保留较多自由度。
📝 Abstract
A central question in modern machine learning is how much a trained model can be compressed without changing its behavior, to reduce the memory, compute and energy required to deploy it. To study this, we quantify functional degeneracy through the behavioral recovery rank, defined as the number of leading behavioral-Hessian eigendirections required to recover a trained model's performance. Using the behavioral recovery rank as a geometric benchmark for compression, we find that structural and magnitude pruning retain more degrees of freedom, even after the task is saturated. This gap suggests that functional redundancy is distributed across parameter directions and is not exposed by individual weights or neurons.
Innovation
Methods, ideas, or system contributions that make the work stand out.
functional degeneracy
behavioral recovery rank
structural and magnitude pruning
🔎 Similar Papers
2023-05-10ACM Computing SurveysCitations: 60
