NeutronSparse: Coordinating Heterogeneous Engines for Sparse Matrix Multiplication on NPUs

2026-06-21

AI Summary

This work addresses the inefficiency of sparse matrix-matrix multiplication (SpMM) on neural processing units (NPUs), which suffers from insufficient coordination among heterogeneous compute units and suboptimal data scheduling, resulting in significantly lower performance compared to GPUs. To overcome these limitations, the authors propose NeutronSparse, a novel framework that introduces, for the first time, a coordination-first SpMM execution strategy. NeutronSparse achieves adaptive load balancing and enhanced data reuse through sparsity-aware collaboration between heterogeneous engines, locality-aware data tiling, and a tile-based execution model. Evaluated on the Ascend 910B NPU, NeutronSparse delivers speedups of 1.26–7.78× over NPU baselines and outperforms state-of-the-art GPU libraries such as cuSPARSE on the NVIDIA A100 by 1.03–3.07×, thereby breaking the performance bottleneck of sparse computation on NPUs.

Abstract

Sparse matrix-matrix multiplication (SpMM) is a fundamental data operation for large-scale sparse data processing. With NPUs increasingly deployed in data centers for their performance and energy efficiency, accelerating SpMM on these platforms is a natural choice. However, high-performance SpMM on NPUs poses a data management challenge, as irregular sparsity demands efficient data organization and scheduling. On Ascend 910B, the official MindSpore implementation achieves only 36.3% of the performance of GPU-based sparse libraries such as cuSPARSE on NVIDIA A100. To this end, we conduct an in-depth architectural analysis of SpMM execution on NPUs versus GPU and identify that the key performance bottleneck for SpMM on NPUs lies in the lack of efficient coordination across heterogeneous compute units under tile-based execution model. Therefore, we propose NeutronSparse, a coordination-first SpMM framework for NPUs. NeutronSparse integrates two key techniques: (i) Sparsity-aware coordination of heterogeneous engines, which adaptively partitions and balances workloads between heterogeneous compute units to keep them busy, and (ii) Locality-aware tile orchestrating, which reorganizes and reuses data tiles to reduce redundant computation and memory movement overhead. Evaluations on Ascend 910B show that NeutronSparse achieves 1.26x-7.78x speedup over NPU baselines and 1.03x-3.07x speedup over leading GPU libraries on NVIDIA A100, revealing untapped potential of NPUs for sparse computation.

Problem

• Sparse Matrix Multiplication
• NPU
• Heterogeneous Engines
• Data Sparsity
• Performance Bottleneck

Innovation

• SpMM
• NPU
• heterogeneous coordination
• sparsity-aware scheduling
• tile orchestration