SpSYRK: Half the Work in Distributed Sparse Matrix Multiplication

2026-08-10

AI Summary

This work addresses the inefficiency of existing distributed sparse matrix multiplication algorithms, which fail to exploit symmetry and consequently incur redundant computation and communication in sparse symmetric rank-k updates (SYRK). The paper presents the first distributed sparse SYRK algorithm that explicitly leverages the symmetry of both input and output matrices by computing and storing only the lower triangular part, thereby eliminating the need for explicit transposition. The approach integrates triangular block partitioning over a process grid, optimized local sparse multiplications, and reordered communication patterns, guided by an arithmetic intensity–driven cost model. Evaluated on 32 nodes of the Perlmutter supercomputer, the proposed method achieves up to 2× speedup over an optimized Sparse SUMMA implementation, with pronounced benefits in compute-intensive regimes and strong scalability.

Abstract

The symmetric rank-$k$ update (SYRK), $\C = \A\A^\top$, computes the dot product between each pair of rows of $\A$, producing the Gram matrix $\C$. Its sparse variant underpins similarity search in machine learning, graph analytics, and genomics, including Jaccard similarity on datasets too large for a single node. Yet, despite the symmetry in its inputs and outputs, existing distributed sparse matrix multiplication algorithms, such as Sparse SUMMA, treat sparse SYRK as generic multiplication, leaving performance untapped. In this paper, we present distributed sparse SYRK approaches that leverage symmetry. The approach partitions the off-diagonal blocks of the output between the upper and lower triangular portions of the process grid and computes only the lower-triangular part of each diagonal block, reducing per-process communication and computation compared with state-of-the-art distributed SpGEMM. A second variant reorders communication to avoid materializing $\A^\top$. On 32 nodes of the Perlmutter supercomputer, the algorithm achieves a $2\times$ speedup over an optimized Sparse SUMMA on matrices where local multiplication dominates the runtime; the advantage narrows on communication-bound inputs, a dependence the cost model predicts from the arithmetic intensity. Our variant rectifies this and consistently achieves superior scaling at high process counts. The approach is a drop-in replacement for any application computing $\C = \A \A^\top$ via a distributed SpGEMM routine, and its triangular output can be consumed directly by subsequent operations, reducing both compute and memory footprint.

Problem

• sparse SYRK
• distributed sparse matrix multiplication
• symmetry exploitation
• SpGEMM
• Gram matrix

Innovation

• sparse SYRK
• symmetry exploitation
• distributed SpGEMM
• communication avoidance
• triangular matrix computation