SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training

2026-05-26

AI Summary

This work addresses the slow convergence of dynamic sparse training (DST) caused by gradient distribution imbalances induced by batch normalization (BN), which hinders DST from matching the efficiency of dense training. For the first time, the study systematically uncovers the interaction mechanisms among BN, sparse architectures, and DST. To mitigate the gradient skew introduced by BN, the authors propose SparseOpt, a sparsity-aware optimizer that employs a gradient correction strategy. Experimental results on ResNet models trained on CIFAR-100 and ImageNet demonstrate that SparseOpt significantly accelerates convergence, enhances generalization performance, and effectively narrows the efficiency gap between DST and dense training.

Abstract

Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology. Despite the promise of reduced computation, DST methods converge significantly slower than dense training, often requiring comparable training time to achieve similar accuracy. We demonstrate both analytically and empirically that Batch Normalization (BN) adversely affects sparse training, and propose SparseOpt, a sparsity-aware optimizer, to address this. Experiments on ResNet models across CIFAR-100 and ImageNet demonstrate consistently faster convergence and improved generalization with our proposed method. Our work highlights the limitations of current normalization layers in sparse training and provides the first systematic study of the interaction between Batch Normalization, sparse layers, and DST, taking a significant step toward making DST practically competitive with dense training.

Problem

• Dynamic Sparse Training
• Batch Normalization
• Gradient Skew
• Sparse Training
• Convergence

Innovation

• Sparse Training
• Batch Normalization
• Gradient Skew
• Dynamic Sparse Training
• SparseOpt