nvshmem

This work addresses the lack of a unified understanding of NVSHMEM’s system-level design and behavior, which has hindered its efficient use in GPU communication. It establishes NVSHMEM as the pioneering device-side symmetric memory programming model and provides a comprehensive analysis of its programming abstractions, implementation mechanisms, and performance characteristics. The study focuses on symmetric memory management, GPU-initiated one-sided communication, and device-side collective operations, empirically evaluating these features using the DeepEP sparse deep learning workload. The findings reveal NVSHMEM’s critical role and inherent design trade-offs in fine-grained, GPU-driven communication, demonstrate its ability to approach hardware performance limits, and solidify its position as a foundational component for GPU communication systems. Furthermore, the work identifies promising directions for runtime-level optimizations.

This work addresses the lack of portable memory semantics for GPU architectures in existing OpenSHMEM implementations, which has led to fragmented vendor-specific approaches. To resolve this, the authors propose a lightweight, backward-compatible auxiliary specification that, for the first time, defines a vendor-neutral GPU-scoped memory space abstraction along with a complementary capability query mechanism. This unified framework harmonizes the semantics of remote memory access (RMA), atomic operations, synchronization, and collective operations under both host-initiated and device-initiated execution models. Built upon the Partitioned Global Address Space (PGAS) paradigm, the proposal is accompanied by a prototype implementation deployable across multiple GPU vendors, thereby establishing the first practical foundation and standardization pathway for GPU extensions to the OpenSHMEM specification.

This work addresses the performance limitations of current GPU communication APIs, which either rely on CPU involvement or impose substantial synchronization overhead, thereby constraining the efficiency of machine learning and high-performance computing applications. The authors propose and implement a novel MPI-based GPU communication abstraction that, for the first time, enables fully CPU-bypassed GPU-to-GPU communication within the MPI framework and natively supports halo exchange primitives such as gather and scatter. By integrating MPI extensions, HPE Slingshot 11 network hardware, and the Cabana/Kokkos portable programming model, the design achieves a 50% reduction in medium-message latency and demonstrates a 28% improvement in halo exchange performance at strong scale on 8,192 GPUs on the Frontier supercomputer.

Modern AI workloads—particularly Mixture-of-Experts (MoE)—demand ultra-low-latency, fine-grained, GPU-native inter-GPU communication, which existing CPU-coordinated approaches fail to meet. Method: This paper introduces GPU-Initiated Networking (GIN), a novel communication architecture wherein GPU kernels directly initiate communication without CPU intervention. GIN features a three-layer design: an NCCL Core host interface, device-side callable CUDA APIs, and a dual-mode network plugin supporting both direct NIC access via DOCA GPUNetIO and RDMA-compatible proxy mode. Contribution/Results: GIN is the first production-grade communication library to enable fully device-initiated, fine-grained communication while maintaining seamless integration with the NCCL ecosystem. Evaluations on MoE workloads such as DeepEP demonstrate substantial latency reduction in inter-GPU communication, with full backward compatibility with NCCL collective primitives and existing infrastructure.

To address the limited strong scaling of GROMACS on heterogeneous supercomputers—caused by MPI communication latency (sub-millisecond) and low GPU utilization—this work proposes a novel GPU-kernel-initiated communication paradigm. We redesign the domain-decomposition halo exchange algorithm to tightly integrate data packing and NVSHMEM communication within GPU kernels, enabling aggressive overlap of computation and communication. Leveraging NVLink’s asynchronous copy engine and kernel fusion techniques, we bypass the CPU-centric MPI communication path, thereby significantly reducing latency and improving bandwidth efficiency. Experimental results demonstrate 1.5× speedup on single-node NVLink systems, 2× on multi-node NVLink clusters, and 1.3× on hybrid NVLink–InfiniBand networks. This approach effectively overcomes the heterogeneous scalability bottleneck of traditional MPI-based molecular dynamics simulations.

This work addresses the inability of GPUs in current high-performance computing (HPC) systems to autonomously initiate cross-node communication, particularly the lack of an efficient, low-overhead GPU-driven communication mechanism on OFI-based interconnects such as Slingshot, alongside inefficient NIC resource reclamation. The authors propose GICC, a runtime system that, for the first time on OFI architectures, enables GPUs to directly trigger NIC operations without host intervention, facilitating fine-grained overlap of computation and communication. GICC also introduces an asynchronous, lock-free, lightweight resource reclamation mechanism. Experimental results demonstrate a 229× reduction in coordination latency and a 25% improvement in weak scaling efficiency on Slingshot; on InfiniBand, it achieves 1.95× lower Put latency compared to NVSHMEM. In an industrial-scale stencil application, GICC attains 42% parallel efficiency, significantly outperforming MPI’s 35.4%.

This work addresses the challenge of graph-based approximate nearest neighbor search (Graph ANNS) when index sizes exceed the memory capacity of a single GPU, a scenario where existing CPU-centric remote I/O approaches struggle to meet the dynamic and irregular data access patterns inherent in GPU-driven graph traversal. To overcome this limitation, the authors propose the first GPU-centric NVMe-oF remote I/O architecture, which fully offloads query execution, page fault handling, and recovery decisions to the GPU, relegating the CPU to a mere transport proxy. They further introduce a staged remote operation mechanism coupled with an ANNS-specific scheduling policy to effectively overlap graph traversal with remote page loading. Experimental results on the SIFT1M DiskANN workload demonstrate that the proposed system achieves 1.31× speedup over the state-of-the-art remote I/O solution and 3.73× speedup compared to a direct-attached remote page caching approach.

This work addresses the high intra-node multi-GPU communication overhead in MPI-based high-performance computing by seamlessly integrating CUDA Graphs into the UCX communication framework for the first time. By co-scheduling multiple communication pathways—including NVLink and PCIe—the proposed approach enables unified optimization of point-to-point GPU data transfers. This strategy overcomes the bandwidth limitations inherent in conventional single-path methods such as UCT::CUDA-IPC. Experimental results on a four-GPU node demonstrate a peak bandwidth improvement of up to 2.95× for 512 MB messages, significantly enhancing multi-GPU communication efficiency.

This work addresses the lack of efficient support for dynamic migration of work items—such as rays—across GPUs in multi-node, multi-GPU data-parallel computing. The authors propose RaFI, a software framework built on CUDA and MPI, which introduces, for the first time, a unified interface enabling GPU kernels to succinctly forward work items to other GPUs while automatically managing the underlying communication and data transfers. By abstracting away the complexities of coordinated CUDA-MPI programming, RaFI significantly simplifies the development of multi-GPU collaborative applications. Empirical evaluation across several use cases demonstrates that the framework not only eases programming but also maintains high performance and strong scalability.

This study addresses the low utilization of modern GPU computing resources by systematically evaluating the performance, energy efficiency, and resource isolation characteristics of NVIDIA’s Multi-Process Service (MPS) and Multi-Instance GPU (MIG) technologies under concurrent application workloads. The experiments reveal a critical trade-off between MPS’s scheduling flexibility and MIG’s hardware-level isolation: MPS can improve performance by up to 30% and reduce energy consumption by approximately 20% in the absence of memory contention, yet suffers a 30% performance degradation under contention; MIG effectively mitigates resource contention but is constrained by its rigid configuration options and higher overhead. These findings provide empirical foundations for optimizing GPU co-execution strategies driven by application-specific workload characteristics.