Automatic Pruning via Structured Lasso with Class-wise Information

2025-02-13

AI Summary

Existing structured pruning methods neglect inter-class statistical disparities, leading to loss of discriminative information. To address this, we propose a semantic-aware structured Lasso pruning framework grounded in the information bottleneck (IB) theory—marking the first integration of class-conditional feature statistics into structured sparsity constraints. Our framework comprises two adaptive variants: graph-guided (sGLP-IB) and tree-guided (sTLP-IB) pruning, both leveraging structural regularization (graph or hierarchical tree) to enforce class-sensitive channel selection and explicitly preserve inter-class separability during compression. On CIFAR-10 with VGG-16, our method achieves 85% parameter reduction, 61% FLOPs reduction, and a 0.14% accuracy gain; on ImageNet with ResNet-50, it reduces parameters by 55% while incurring only a 0.03% top-1 accuracy drop—substantially outperforming state-of-the-art approaches.

Technology Category

• Machine Learning: Structured Learning
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Most pruning methods concentrate on unimportant filters of neural networks. However, they face the loss of statistical information due to a lack of consideration for class-wise data. In this paper, from the perspective of leveraging precise class-wise information for model pruning, we utilize structured lasso with guidance from Information Bottleneck theory. Our approach ensures that statistical information is retained during the pruning process. With these techniques, we introduce two innovative adaptive network pruning schemes: sparse graph-structured lasso pruning with Information Bottleneck ( extbf{sGLP-IB}) and sparse tree-guided lasso pruning with Information Bottleneck ( extbf{sTLP-IB}). The key aspect is pruning model filters using sGLP-IB and sTLP-IB to better capture class-wise relatedness. Compared to multiple state-of-the-art methods, our approaches demonstrate superior performance across three datasets and six model architectures in extensive experiments. For instance, using the VGG16 model on the CIFAR-10 dataset, we achieve a parameter reduction of 85%, a decrease in FLOPs by 61%, and maintain an accuracy of 94.10% (0.14% higher than the original model); we reduce the parameters by 55% with the accuracy at 76.12% using the ResNet architecture on ImageNet (only drops 0.03%). In summary, we successfully reduce model size and computational resource usage while maintaining accuracy. Our codes are at https://anonymous.4open.science/r/IJCAI-8104.

Problem

• Improving neural network pruning with class-wise data
• Retaining statistical information during pruning process
• Introducing adaptive pruning schemes sGLP-IB and sTLP-IB

Innovation

• Utilizes structured lasso for pruning
• Integrates Information Bottleneck theory
• Develops sGLP-IB and sTLP-IB schemes