nvidia technologies

Modern AI workloads—particularly Mixture-of-Experts (MoE)—demand ultra-low-latency, fine-grained, GPU-native inter-GPU communication, which existing CPU-coordinated approaches fail to meet. Method: This paper introduces GPU-Initiated Networking (GIN), a novel communication architecture wherein GPU kernels directly initiate communication without CPU intervention. GIN features a three-layer design: an NCCL Core host interface, device-side callable CUDA APIs, and a dual-mode network plugin supporting both direct NIC access via DOCA GPUNetIO and RDMA-compatible proxy mode. Contribution/Results: GIN is the first production-grade communication library to enable fully device-initiated, fine-grained communication while maintaining seamless integration with the NCCL ecosystem. Evaluations on MoE workloads such as DeepEP demonstrate substantial latency reduction in inter-GPU communication, with full backward compatibility with NCCL collective primitives and existing infrastructure.

NCCL—the de facto standard library for high-performance collective communication in GPU clusters—lacks transparency in its internal protocol selection, channel orchestration, and cross-node memory movement mechanisms, hindering systematic performance analysis and bottleneck identification. Method: We present the first systematic reverse-engineering of NCCL’s multi-tier communication architecture, integrating trace-driven modeling with fine-grained analysis of its three core protocols (Simple, LL, LL128) to uncover the dynamic scheduling logic of ring and tree algorithms—and their associated data movement strategies—under realistic AI training workloads. Contribution/Results: Based on these insights, we develop ATLAHS, a reproducible, industrial-grade simulation toolchain that accurately models NCCL’s communication behavior, enabling precise performance prediction and root-cause bottleneck diagnosis. Our work establishes a verifiable theoretical foundation and practical toolset for designing high-performance communication libraries, optimizing AI training systems, and enabling hardware-software co-tuning.

In resource-constrained countries—such as Nepal—deploying deep learning infrastructure faces significant challenges, including high capital expenditure, excessive power consumption, and weak cybersecurity. To address these, this work designs and deploys a low-cost, energy-efficient GPU cluster comprising one management node and three compute nodes, each equipped with an NVIDIA GTX 1650 GPU. The system integrates Slurm for job scheduling, Anaconda for multi-environment management, NFS for shared storage, and fail2ban for intrusion prevention. This represents the first full-stack, lightweight GPU cluster deployment tailored for AI and scientific computing in such settings. Empirical evaluation demonstrates stable execution of deep learning training and scientific simulations; SSH brute-force attack mitigation achieves 98% success rate; and per-unit computational cost is reduced by 67% compared to commercial alternatives. The architecture exhibits strong scalability and practical transferability to similar low-resource contexts.

To address flow collisions and degraded collective communication performance in AI training caused by inflexible RDMA network transport, this paper proposes UCCL—a scalable GPU-network software transport layer. Our approach adopts a hardware-software co-designed, scalable architecture: it decouples data and control planes on RDMA NICs, migrating control logic to the CPU to enable flexible protocol evolution; and introduces a multipath scheduling mechanism with traffic-balancing algorithms to avoid single-path congestion and overcome inherent hardware limitations. Experimental evaluation demonstrates that UCCL achieves 3.3× higher ML collective communication throughput compared to state-of-the-art industrial solutions, significantly accelerating large-model training.

This study addresses the GPU computational efficiency bottleneck in deep and machine learning. Methodologically, it proposes a task-aware GPU parallel architecture adaptation framework that systematically integrates CUDA stream-based concurrency, dynamic parallelism, and heterogeneous hardware (FPGA/TPU/ASIC) co-selection—implemented via deep integration into PyTorch, TensorFlow, and XGBoost. Its core contribution lies in establishing a transferable GPU optimization methodology, transcending model- or library-specific tuning. Experimental evaluation demonstrates 3–8× speedup across representative training and inference workloads. Furthermore, the authors open-source a modular, well-documented GPU optimization practice guide, substantially lowering the barrier to entry for AI practitioners seeking parallelization optimizations.

This work addresses the high intra-node multi-GPU communication overhead in MPI-based high-performance computing by seamlessly integrating CUDA Graphs into the UCX communication framework for the first time. By co-scheduling multiple communication pathways—including NVLink and PCIe—the proposed approach enables unified optimization of point-to-point GPU data transfers. This strategy overcomes the bandwidth limitations inherent in conventional single-path methods such as UCT::CUDA-IPC. Experimental results on a four-GPU node demonstrate a peak bandwidth improvement of up to 2.95× for 512 MB messages, significantly enhancing multi-GPU communication efficiency.

This work addresses the lack of efficient support for dynamic migration of work items—such as rays—across GPUs in multi-node, multi-GPU data-parallel computing. The authors propose RaFI, a software framework built on CUDA and MPI, which introduces, for the first time, a unified interface enabling GPU kernels to succinctly forward work items to other GPUs while automatically managing the underlying communication and data transfers. By abstracting away the complexities of coordinated CUDA-MPI programming, RaFI significantly simplifies the development of multi-GPU collaborative applications. Empirical evaluation across several use cases demonstrates that the framework not only eases programming but also maintains high performance and strong scalability.

Existing in-network computing approaches based on NVLink SHARP suffer from inefficient overlap between communication and computation due to mismatches in their memory semantics, leading to suboptimal resource utilization in multi-GPU systems. This work proposes CAIS, the first compute-aware in-network computing framework, which aligns the memory semantics of large language model (LLM) computation kernels with communication through three key innovations: a compute-aware instruction set and microarchitectural extensions, a coalescing-aware thread block coordination mechanism, and a graph-level dataflow optimizer. By breaking away from conventional communication-centric designs, CAIS achieves significant performance gains—accelerating LLM training by 1.38× over the state-of-the-art NVLS-based approach and by 1.61× compared to the T3 baseline without NVLS.

This study addresses the lack of native GPU monitoring tools in R for computational biology, where users traditionally rely on external command-line utilities. To bridge this gap, we introduce CudaMon, an R package that leverages the NVIDIA Management Library (NVML) interface to provide real-time monitoring of GPU utilization, memory usage, temperature, and power consumption. CudaMon seamlessly integrates data export and visualization capabilities into standard R workflows, significantly enhancing the efficiency and reproducibility of GPU-accelerated analyses. In a large-scale single-cell RNA-seq analysis involving millions of brain cells, the tool successfully identified stages—such as PCA and UMAP—where GPU utilization exceeded 90%, while also uncovering performance bottlenecks during data management phases.

This work addresses the lack of transparency in NVIDIA’s closed-source user-space driver, which obscures the translation of CUDA API calls into hardware commands and impedes understanding of GPU behavior and performance attribution. The authors propose a novel approach that requires no modification to the proprietary driver, instead leveraging an open-source kernel driver, memory-mapped path instrumentation, and hardware watchpoints on the GPU’s doorbell registers to capture and reconstruct the complete low-level command stream with unprecedented accuracy. This methodology reveals the true DMA patterns and performance characteristics of CUDA data transfers and demonstrates that the low overhead of CUDA Graphs stems from their streamlined and efficient command submission mechanism. By significantly enhancing the interpretability of GPU runtime behavior, this approach establishes a new paradigm for middleware analysis and hardware-software co-design.