ucx

This work addresses the performance limitations of current GPU communication APIs, which either rely on CPU involvement or impose substantial synchronization overhead, thereby constraining the efficiency of machine learning and high-performance computing applications. The authors propose and implement a novel MPI-based GPU communication abstraction that, for the first time, enables fully CPU-bypassed GPU-to-GPU communication within the MPI framework and natively supports halo exchange primitives such as gather and scatter. By integrating MPI extensions, HPE Slingshot 11 network hardware, and the Cabana/Kokkos portable programming model, the design achieves a 50% reduction in medium-message latency and demonstrates a 28% improvement in halo exchange performance at strong scale on 8,192 GPUs on the Frontier supercomputer.

This work addresses the significant overhead of collective communication in large model training, where existing compute-communication overlap techniques require extensive code modifications and struggle to support complex parallelism strategies such as tensor and expert parallelism. To overcome these limitations, the authors propose CCCL, the first collective communication library that natively integrates compression without user intervention. CCCL leverages on-GPU compression, fuses communication with computation, and deeply integrates with NCCL to natively support operations like AllReduce and AllToAll while eliminating the data concatenation phase inherent in conventional compression approaches, thereby substantially reducing memory access overhead. Experimental results demonstrate that CCCL achieves up to a 10.1% improvement in end-to-end throughput under vLLM PD-decoupled workloads and up to 30% higher communication throughput in microbenchmarks.

To address redundant data movement, buffer overhead, and high communication latency in MPI caused by explicit memory copying in distributed HPC systems, this paper proposes the first cache-coherent shared-memory MPI communication paradigm based on CXL 3.2. Leveraging hardware-supported cross-host virtual address space direct mapping, it enables message pointer passing instead of data copying, thereby eliminating traditional MPI’s memory copy and serialization overheads. We design and implement an end-to-end co-designed system integrating a CXL controller, an FPGA-based multi-host emulation platform, and a customized software stack to support zero-copy, low-latency MPI runtime execution. Experimental evaluation on representative HPC benchmarks demonstrates up to 47% reduction in communication latency and a 2.1× improvement in bandwidth utilization, significantly enhancing scalability and energy efficiency for large-scale applications.

This work addresses the challenges of GPU memory and interconnect bandwidth limitations in multi-node large language model (LLM) training and inference, where conventional RDMA-based communication incurs high overhead. The authors propose, for the first time, a cross-node GPU collective communication library leveraging CXL-based shared memory pools, replacing RDMA with a memory-centric architecture to effectively tackle key issues such as synchronization, data interleaving, and communication parallelization. Their approach achieves speedups of 1.34×, 1.84×, 1.94×, and 1.04× on AllGather, Broadcast, Gather, and Scatter operations, respectively, and delivers an end-to-end 1.11× acceleration in LLM training while reducing hardware costs by up to 2.75×.

To address three critical limitations of NCCL in large-scale GPU training—inefficient peer-to-peer (P2P) communication, poor fault tolerance against RoCE NIC (RNIC) port failures, and difficulty in observing transient collective communication anomalies—this paper proposes ICCL, a novel communication library that is efficient, reliable, and highly observable. ICCL’s key innovations are: (1) offloading P2P communication from GPUs to CPU threads to free GPU SM resources; (2) a primary-backup queue pair (QP) mechanism enabling millisecond-level RNIC failover; and (3) microsecond-granularity sliding-window network monitoring for precise detection of transient anomalies. Experiments show ICCL improves P2P throughput by 23.4% and reduces latency by 28.5% over NCCL, yielding a 6.02% end-to-end training throughput gain. ICCL has operated stably in production for several months and is open-sourced.

Existing tools struggle to provide fine-grained tracing of the UCX communication layer and are often confined to specific MPI implementations, hindering effective correlation between high-level MPI calls and low-level device communication behavior. To address this limitation, this work proposes ucTrace—the first general-purpose tool enabling fine-grained communication analysis at the UCX level. By jointly tracing process- and device-level events, ucTrace accurately maps communication operations across host, GPU, and NIC back to their originating MPI functions and offers interactive visualization. Crucially, ucTrace is independent of any particular MPI library and supports heterogeneous environments such as GPU-accelerated systems. It has been successfully applied to diverse HPC workloads, including point-to-point communication optimization, Allreduce performance comparison across multiple MPI libraries, communication profiling in linear solvers, NUMA binding evaluation, and large-scale GPU-accelerated GROMACS simulations.

This work addresses the significant communication bottleneck in multi-GPU training caused by the serial execution of computation and communication. The authors propose a portable runtime mechanism that requires no modifications to vendor libraries or kernels. By dynamically controlling on-chip resource occupancy of compute kernels, elevating the scheduling priority of communication streams, and leveraging shared memory for compute footprint management and cross-GPU resource coordination, the approach effectively enables concurrent execution of computation and collective communication. Evaluated on NVIDIA A40, A100, H100, and AMD MI250X GPUs, the method reduces end-to-end training time by up to 25.5%.

This work addresses the inability of GPUs in current high-performance computing (HPC) systems to autonomously initiate cross-node communication, particularly the lack of an efficient, low-overhead GPU-driven communication mechanism on OFI-based interconnects such as Slingshot, alongside inefficient NIC resource reclamation. The authors propose GICC, a runtime system that, for the first time on OFI architectures, enables GPUs to directly trigger NIC operations without host intervention, facilitating fine-grained overlap of computation and communication. GICC also introduces an asynchronous, lock-free, lightweight resource reclamation mechanism. Experimental results demonstrate a 229× reduction in coordination latency and a 25% improvement in weak scaling efficiency on Slingshot; on InfiniBand, it achieves 1.95× lower Put latency compared to NVSHMEM. In an industrial-scale stencil application, GICC attains 42% parallel efficiency, significantly outperforming MPI’s 35.4%.

Training large language models on heterogeneous clusters with mixed-vendor hardware is hindered by existing collective communication frameworks that struggle to efficiently accommodate diverse hardware characteristics, resulting in high communication overhead and low bandwidth utilization. This work proposes HetCCL, a novel communication framework that eliminates host-device memory copies through heterogeneous point-to-point transfers and control-plane CPU offloading. It introduces a boundary communicator mechanism to enable vendor-agnostic reduction operations and employs a hierarchical topology abstraction to optimize cross-device data movement. Integrated with multi-vendor communication libraries, HetCCL achieves 17–19× higher communication bandwidth than Gloo across four heterogeneous configurations and accelerates end-to-end training by up to 16.9% per iteration.

This work addresses the high intra-node multi-GPU communication overhead in MPI-based high-performance computing by seamlessly integrating CUDA Graphs into the UCX communication framework for the first time. By co-scheduling multiple communication pathways—including NVLink and PCIe—the proposed approach enables unified optimization of point-to-point GPU data transfers. This strategy overcomes the bandwidth limitations inherent in conventional single-path methods such as UCT::CUDA-IPC. Experimental results on a four-GPU node demonstrate a peak bandwidth improvement of up to 2.95× for 512 MB messages, significantly enhancing multi-GPU communication efficiency.