Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning

2024-11-20

arXiv.org

AI Summary

Deep neural networks suffer from severe accuracy degradation under extreme sparsity (99.90%–99.99%) due to gradient instability and layer collapse. To address this, we propose Extreme Adaptive Sparse Training (EAST), a novel sparse training framework. EAST introduces three synergistic mechanisms: (i) dynamic ReLU phase switching to enhance gradient flow robustness; (ii) intra-residual structured weight sharing to alleviate parameter scarcity; and (iii) cyclically evolving sparsity levels and sparsity patterns for adaptive, stability-preserving updates. Crucially, EAST requires neither retraining nor dense initialization. On CIFAR-10/100 and ImageNet, ResNet-34 and ResNet-50 trained with EAST achieve near-dense-model accuracy at 99.99% sparsity—substantially outperforming state-of-the-art sparse training methods.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Pruning of deep neural networks has been an effective technique for reducing model size while preserving most of the performance of dense networks, crucial for deploying models on memory and power-constrained devices. While recent sparse learning methods have shown promising performance up to moderate sparsity levels such as 95% and 98%, accuracy quickly deteriorates when pushing sparsities to extreme levels. Obtaining sparse networks at such extreme sparsity levels presents unique challenges, such as fragile gradient flow and heightened risk of layer collapse. In this work, we explore network performance beyond the commonly studied sparsities, and propose a collection of techniques that enable the continuous learning of networks without accuracy collapse even at extreme sparsities, including 99.90%, 99.95% and 99.99% on ResNet architectures. Our approach combines 1) Dynamic ReLU phasing, where DyReLU initially allows for richer parameter exploration before being gradually replaced by standard ReLU, 2) weight sharing which reuses parameters within a residual layer while maintaining the same number of learnable parameters, and 3) cyclic sparsity, where both sparsity levels and sparsity patterns evolve dynamically throughout training to better encourage parameter exploration. We evaluate our method, which we term Extreme Adaptive Sparse Training (EAST) at extreme sparsities using ResNet-34 and ResNet-50 on CIFAR-10, CIFAR-100, and ImageNet, achieving significant performance improvements over state-of-the-art methods we compared with.

Problem

• Addresses accuracy collapse in deep neural networks at extreme sparsity levels.
• Proposes techniques to maintain performance in highly pruned networks.
• Enables continuous learning without degradation at sparsities up to 99.99%.

Innovation

• Dynamic ReLU phasing for parameter exploration
• Weight sharing within residual layers
• Cyclic sparsity for dynamic training adaptation