WingSpan: Concurrency and Dependence for Sparse and Structured Tensor Compilers

2026-06-18

AI Summary

Existing sparse tensor compilers struggle to achieve safe and efficient parallelization in scenarios involving sparsity in the output or multiple sparse inputs. To address this challenge, this work introduces WingSpan, a sparse tensor language that, for the first time, supports unrestricted combinations of parallel loops and data structures. It also develops a concurrency dependency theory tailored for both sparse and structured tensors, ensuring correctness of parallel execution. By unifying these advances, the proposed framework delivers general-purpose parallel support for sparse tensor programs, matching or exceeding the performance of hand-optimized parallel implementations on key kernels such as sparse general matrix-matrix multiplication (SpGEMM).

Abstract

Sparse tensors represent data that is mostly zero or some other compressible fill pattern. Such datasets can be massive, so optimized tensor algebra libraries and compilers have been developed to exploit these patterns to improve performance. Existing systems, however, frequently lack support for parallelism, especially when outputs are sparse or multiple inputs are sparse. We propose WingSpan, a sparse tensor language enabling unrestricted parallel programming. WingSpan supports arbitrary composition of parallel loops and data structures, matching or exceeding the performance of hand-optimized parallel routines on critical kernels such as SpGEMM. We also introduce a dependence theory for the safety of parallel programs involving sparse tensors and structures beyond sparsity.

Problem

• sparse tensors
• parallelism
• tensor compilers
• dependence analysis
• structured data

Innovation

• sparse tensors
• parallel programming
• dependence analysis
• tensor compilers
• SpGEMM