cuda & multi-gpu systems

NCCL—the de facto standard library for high-performance collective communication in GPU clusters—lacks transparency in its internal protocol selection, channel orchestration, and cross-node memory movement mechanisms, hindering systematic performance analysis and bottleneck identification. Method: We present the first systematic reverse-engineering of NCCL’s multi-tier communication architecture, integrating trace-driven modeling with fine-grained analysis of its three core protocols (Simple, LL, LL128) to uncover the dynamic scheduling logic of ring and tree algorithms—and their associated data movement strategies—under realistic AI training workloads. Contribution/Results: Based on these insights, we develop ATLAHS, a reproducible, industrial-grade simulation toolchain that accurately models NCCL’s communication behavior, enabling precise performance prediction and root-cause bottleneck diagnosis. Our work establishes a verifiable theoretical foundation and practical toolset for designing high-performance communication libraries, optimizing AI training systems, and enabling hardware-software co-tuning.

To address CPU-induced communication bottlenecks in heterogeneous supercomputing, this paper systematically proposes and classifies the “GPU-centric communication” paradigm—fully migrating communication control from the CPU to the GPU to enable zero-CPU-intervention, end-to-end GPU-direct communication. Methodologically, it integrates GPUDirect RDMA, CUDA Graphs, programmable NICs (e.g., BlueField DPUs), and GPU kernel-level communication primitives to construct a GPU-native communication control stack. Key contributions include: (1) the first systematic characterization of hardware requirements—GPU memory coherence, NIC programmability, and kernel-driver support—and software challenges—including synchronization, error handling, and programming model adaptation; and (2) a design blueprint for low-latency, high-throughput, ultra-low-CPU-overhead communication infrastructure tailored for HPC and distributed AI training. (149 words)

This work addresses the significant communication bottleneck in multi-GPU training caused by the serial execution of computation and communication. The authors propose a portable runtime mechanism that requires no modifications to vendor libraries or kernels. By dynamically controlling on-chip resource occupancy of compute kernels, elevating the scheduling priority of communication streams, and leveraging shared memory for compute footprint management and cross-GPU resource coordination, the approach effectively enables concurrent execution of computation and collective communication. Evaluated on NVIDIA A40, A100, H100, and AMD MI250X GPUs, the method reduces end-to-end training time by up to 25.5%.

To address three critical limitations of NCCL in large-scale GPU training—inefficient peer-to-peer (P2P) communication, poor fault tolerance against RoCE NIC (RNIC) port failures, and difficulty in observing transient collective communication anomalies—this paper proposes ICCL, a novel communication library that is efficient, reliable, and highly observable. ICCL’s key innovations are: (1) offloading P2P communication from GPUs to CPU threads to free GPU SM resources; (2) a primary-backup queue pair (QP) mechanism enabling millisecond-level RNIC failover; and (3) microsecond-granularity sliding-window network monitoring for precise detection of transient anomalies. Experiments show ICCL improves P2P throughput by 23.4% and reduces latency by 28.5% over NCCL, yielding a 6.02% end-to-end training throughput gain. ICCL has operated stably in production for several months and is open-sourced.

Distributed deep learning (DL) jobs are highly sensitive to communication latency, yet existing schedulers lack network awareness. To address this, we propose a network-aware GPU cluster scheduler. Our method introduces three key innovations: (1) a job latency-sensitivity–driven GPU resource colocation mechanism, the first of its kind; (2) a network-aware, fine-grained preemption strategy that dynamically prioritizes latency-critical communication; and (3) an adaptive latency timer for automatic, runtime tuning of scheduling parameters. Integrated into a data-driven distributed DL simulation platform, our scheduler achieves up to 69% reduction in end-to-end training time, 83% decrease in average job completion time, and 98% lower communication overhead under high network congestion. Furthermore, it significantly improves GPU resource utilization and overall training efficiency.

This work addresses the significant overhead of collective communication in large model training, where existing compute-communication overlap techniques require extensive code modifications and struggle to support complex parallelism strategies such as tensor and expert parallelism. To overcome these limitations, the authors propose CCCL, the first collective communication library that natively integrates compression without user intervention. CCCL leverages on-GPU compression, fuses communication with computation, and deeply integrates with NCCL to natively support operations like AllReduce and AllToAll while eliminating the data concatenation phase inherent in conventional compression approaches, thereby substantially reducing memory access overhead. Experimental results demonstrate that CCCL achieves up to a 10.1% improvement in end-to-end throughput under vLLM PD-decoupled workloads and up to 30% higher communication throughput in microbenchmarks.

This work addresses the low GPU utilization and performance interference caused by coarse-grained resource allocation under mixed HPC, AI, and data analytics workloads. To bridge the granularity gap between static Multi-Instance GPU (MIG) partitions and the actual resource demands of applications, the authors propose a novel co-design mechanism that integrates static MIG partitioning with fine-grained CPU memory offloading, leveraging cache-coherent NVLink-C2C interconnects for the first time. System-level evaluations across representative real-world applications—including NekRS, LAMMPS, Llama3, and Qiskit—demonstrate that the proposed approach significantly reduces GPU idle time, improves throughput and energy efficiency, and effectively mitigates performance degradation due to resource contention in shared environments.

This work addresses critical reliability and safety issues in NCCL plugins, which execute unverified native code and are prone to task crashes, silent state corruption, and downtime during policy updates. The authors propose a userspace eBPF-based runtime mechanism that seamlessly integrates with NCCL’s existing plugin interface, enabling secure, efficient, and composable communication policy execution without modifying NCCL itself. By leveraging static verification at load time, structured cross-plugin shared maps, and atomic hot-reload capabilities, the system guarantees policy safety, composability, and zero-downtime updates. Experimental results on an 8×NVIDIA B300 GPU system demonstrate that the approach incurs only 80–130 nanoseconds of overhead per decision, completely prevents unsafe behaviors observed in testing, and improves AllReduce throughput by up to 27% across message sizes ranging from 4 to 128 MiB.

This work addresses the lack of efficient support for dynamic migration of work items—such as rays—across GPUs in multi-node, multi-GPU data-parallel computing. The authors propose RaFI, a software framework built on CUDA and MPI, which introduces, for the first time, a unified interface enabling GPU kernels to succinctly forward work items to other GPUs while automatically managing the underlying communication and data transfers. By abstracting away the complexities of coordinated CUDA-MPI programming, RaFI significantly simplifies the development of multi-GPU collaborative applications. Empirical evaluation across several use cases demonstrates that the framework not only eases programming but also maintains high performance and strong scalability.

This work addresses the scalability bottleneck in modern GPU clusters caused by skewed communication traffic, which leads to link congestion and uneven resource utilization. The authors propose NIMBLE, a system that dynamically orchestrates multi-path communication across intra-node NVLink and inter-node networks at runtime. NIMBLE achieves transparent, sequentially consistent load balancing through a capacity-normalized minimum-congestion optimization model, requiring no application modifications and seamlessly integrating with existing communication libraries. It employs a multiplicative weight update algorithm to solve the optimization problem and leverages CUDA-aware GPU kernels to drive an RDMA pipeline, intelligently scheduling intermediate GPUs and NICs for routing. Evaluated on an H100-SXM4 cluster, NIMBLE improves intra-node bandwidth by 2.3× and inter-node throughput by 3.8× over single-path baselines, outperforms NCCL and MPI by up to 5.2× under skewed All-to-Allv workloads, and accelerates MoE large model training by 1.35×.