Adaptive Space-efficient Collectives for Dynamic and Unstructured Sparsity on GPU Platforms

2026-07-06

AI Summary

This work addresses the inefficiency of existing collective communication libraries, such as NCCL, which are optimized solely for dense data and struggle with the format conversion overhead and reduction densification inherent in sparse communication. The authors propose a GPU-oriented, sparsity-aware collective communication algorithm that supports all-gather, reduce-scatter, and all-reduce operations while dynamically adapting to input sparsity levels. A key innovation is the introduction of Pici, a novel bit-vector-based sparse format enabling low-overhead compression and rapid decompression, which adaptively preserves sparsity throughout communication. Experimental results demonstrate significant performance gains: at 99% sparsity, the proposed method achieves speedups of 5.25×, 2.5×, and 2.66× over NCCL for all-gather, reduce-scatter, and all-reduce, respectively.

Abstract

High-performance collective communication primitives are necessary for a variety of high performance computing (HPC) and machine learning (ML) workloads. State-of-the-art collective communication libraries such as NCCL optimize exclusively for dense data. However, when sending sparse data, we can reduce communication volume by not sending zeros. Unfortunately, explicitly handling sparsity introduces challenges such as format conversion overheads and densification during collectives that involve reductions. In this paper, we introduce sparsity-exploiting algorithms for three collectives that address these challenges: all-gather, reduce-scatter, and all-reduce. Our collective implementations are backed by a new bitvector-based format, Pici, designed for low overhead and fast (de)compression at moderate sparsities. Further, our algorithms adapt to the level of sparsity in data, modifying its representation during the course of the collective. At 99% input sparsity, our collectives achieve up to 5.25x, 2.5x, and 2.66x speedups over NCCL for all-gather, reduce-scatter, and all-reduce, respectively.

Problem

• sparse data
• collective communication
• GPU platforms
• sparsity
• HPC

Innovation

• sparsity-exploiting collectives
• adaptive communication
• bitvector format
• Pici
• GPU sparse communication