Dynamic Sparse Training of Diagonally Sparse Networks

2025-06-13

AI Summary

To address the limited hardware acceleration of unstructured sparsity in Dynamic Sparse Training (DST), this paper proposes DynaDiag—the first DST method that maintains a fixed diagonal-sparse structure throughout training. DynaDiag integrates dynamic connection updates with hardware-friendly diagonal sparsity constraints and introduces custom CUDA kernels for efficient sparse forward and backward passes. Evaluated on Vision Transformers (ViTs), DynaDiag achieves full-dense model accuracy at 90% sparsity, accelerating inference and training by 3.13× and 1.59×, respectively—substantially outperforming general-purpose sparse methods while matching the accuracy of unstructured DST approaches. This work is the first to simultaneously achieve high accuracy, computational efficiency, and hardware compatibility under extreme sparsity, thereby unifying these three critical objectives in sparse deep learning.

Technology Category

• Machine Learning: Hardware-aware ML
• Computer Vision: Diffusion Models for Vision
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Recent advances in Dynamic Sparse Training (DST) have pushed the frontier of sparse neural network training in structured and unstructured contexts, matching dense-model performance while drastically reducing parameter counts to facilitate model scaling. However, unstructured sparsity often fails to translate into practical speedups on modern hardware. To address this shortcoming, we propose DynaDiag, a novel structured sparse-to-sparse DST method that performs at par with unstructured sparsity. DynaDiag enforces a diagonal sparsity pattern throughout training and preserves sparse computation in forward and backward passes. We further leverage the diagonal structure to accelerate computation via a custom CUDA kernel, rendering the method hardware-friendly. Empirical evaluations on diverse neural architectures demonstrate that our method maintains accuracy on par with unstructured counterparts while benefiting from tangible computational gains. Notably, with 90% sparse linear layers in ViTs, we observe up to a 3.13x speedup in online inference without sacrificing model performance and a 1.59x speedup in training on a GPU compared to equivalent unstructured layers. Our source code is available at https://github.com/horizon-research/DynaDiag/.

Problem

• Improving hardware efficiency of sparse neural networks
• Enforcing diagonal sparsity for structured computation
• Achieving speedups in training and inference without accuracy loss

Innovation

• Diagonal sparsity pattern for structured DST
• Custom CUDA kernel for hardware acceleration
• Maintains accuracy with computational speedups