🤖 AI Summary
This work addresses the lack of theoretical foundations in deep network pruning by systematically investigating its impact on decision boundaries and establishing interpretability principles. We introduce continuous piecewise-affine (CPA) and max-affine spline theory into pruning analysis for the first time, formulating a rigorous mathematical framework to model pruning dynamics. This framework reveals the intrinsic mechanism behind the “early-bird ticket” phenomenon and yields principled, layer-wise and global pruning criteria grounded in activation pattern characterization and network architecture properties—without relying on empirical heuristics. Experiments demonstrate that our approach matches or surpasses state-of-the-art pruning methods in both compression ratio and accuracy, while providing transparent, theory-backed explanations of pruning behavior. By bridging geometric functional analysis with neural network sparsification, this work advances pruning research from an empirical paradigm toward a principle-driven, mathematically grounded one.
📝 Abstract
In this paper, we study the importance of pruning in Deep Networks (DNs) and the yin&yang relationship between (1) pruning highly overparametrized DNs that have been trained from random initialization and (2) training small DNs that have been"cleverly"initialized. As in most cases practitioners can only resort to random initialization, there is a strong need to develop a grounded understanding of DN pruning. Current literature remains largely empirical, lacking a theoretical understanding of how pruning affects DNs' decision boundary, how to interpret pruning, and how to design corresponding principled pruning techniques. To tackle those questions, we propose to employ recent advances in the theoretical analysis of Continuous Piecewise Affine (CPA) DNs. From this perspective, we will be able to detect the early-bird (EB) ticket phenomenon, provide interpretability into current pruning techniques, and develop a principled pruning strategy. In each step of our study, we conduct extensive experiments supporting our claims and results; while our main goal is to enhance the current understanding towards DN pruning instead of developing a new pruning method, our spline pruning criteria in terms of layerwise and global pruning is on par with or even outperforms state-of-the-art pruning methods.
