One-cycle Structured Pruning with Stability Driven Structure Search

2025-01-23

AI Summary

To address the high computational overhead and procedural redundancy in structured pruning, this paper proposes a single-cycle structured pruning framework that unifies pretraining, pruning, and fine-tuning into one end-to-end differentiable training pass. Our method introduces two key innovations: (1) early subnetwork search guided by group-wise significance scoring and structural sparsity regularization; and (2) a stability-driven pruning timing criterion—novelly leveraging inter-epoch subnetwork similarity to automatically identify stable pruning points. Evaluated on CIFAR-10/100 and ImageNet, our approach achieves superior accuracy over state-of-the-art structured pruning methods while reducing total training time by up to 3.2×. It is architecture-agnostic, supporting diverse mainstream models including VGG, ResNet, MobileNet, and Vision Transformers (ViTs).

Technology Category

• Machine Learning: Structured Learning
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Bridging structured and unstructured data
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Privacy-enhancing technologies

Abstract

Existing structured pruning typically involves multi-stage training procedures that often demand heavy computation. Pruning at initialization, which aims to address this limitation, reduces training costs but struggles with performance. To address these challenges, we propose an efficient framework for one-cycle structured pruning without compromising model performance. In this approach, we integrate pre-training, pruning, and fine-tuning into a single training cycle, referred to as the `one cycle approach'. The core idea is to search for the optimal sub-network during the early stages of network training, guided by norm-based group saliency criteria and structured sparsity regularization. We introduce a novel pruning indicator that determines the stable pruning epoch by assessing the similarity between evolving pruning sub-networks across consecutive training epochs. Also, group sparsity regularization helps to accelerate the pruning process and results in speeding up the entire process. Extensive experiments on datasets, including CIFAR-10/100, and ImageNet, using VGGNet, ResNet, MobileNet, and ViT architectures, demonstrate that our method achieves state-of-the-art accuracy while being one of the most efficient pruning frameworks in terms of training time. The source code will be made publicly available.

Problem

• Model Pruning
• Computational Efficiency
• Structured Pruning

Innovation

• One-Cycle Pruning Method
• Efficiency Optimization
• Performance Preservation