graphics processing units (gpus)

This study addresses the GPU computational efficiency bottleneck in deep and machine learning. Methodologically, it proposes a task-aware GPU parallel architecture adaptation framework that systematically integrates CUDA stream-based concurrency, dynamic parallelism, and heterogeneous hardware (FPGA/TPU/ASIC) co-selection—implemented via deep integration into PyTorch, TensorFlow, and XGBoost. Its core contribution lies in establishing a transferable GPU optimization methodology, transcending model- or library-specific tuning. Experimental evaluation demonstrates 3–8× speedup across representative training and inference workloads. Furthermore, the authors open-source a modular, well-documented GPU optimization practice guide, substantially lowering the barrier to entry for AI practitioners seeking parallelization optimizations.

This work addresses the significant communication bottleneck in multi-GPU training caused by the serial execution of computation and communication. The authors propose a portable runtime mechanism that requires no modifications to vendor libraries or kernels. By dynamically controlling on-chip resource occupancy of compute kernels, elevating the scheduling priority of communication streams, and leveraging shared memory for compute footprint management and cross-GPU resource coordination, the approach effectively enables concurrent execution of computation and collective communication. Evaluated on NVIDIA A40, A100, H100, and AMD MI250X GPUs, the method reduces end-to-end training time by up to 25.5%.

This study addresses the inequities in AI accessibility arising from imbalanced computational resource allocation, high energy consumption, and hardware barriers in large-scale deep learning training. Conducting systematic benchmarking on Intel Xeon CPUs and NVIDIA Tesla T4 GPUs across TensorFlow and PyTorch frameworks, we evaluate four representative models—Conv6, VGG16, ResNet18, and CycleGAN. For the first time, we jointly analyze model complexity, framework-level optimizations, and hardware platforms to quantify GPU acceleration benefits, revealing that lightweight models like Conv6 achieve the most significant speedup (up to 246× in training). We also find that TensorFlow’s operator fusion reduces inference latency by approximately 15%. Using polynomial regression, we project GPU memory requirements through 2025 and argue that shared GPU infrastructure is a critical pathway toward democratizing AI and ensuring equitable research opportunities.

This work addresses the challenge of automatically generating efficient GPU kernels for large language models (LLMs). We introduce KernelBench, the first open-source benchmark targeting realistic machine learning operators—comprising 250 PyTorch operators—and jointly evaluating functional correctness and measured speedup. We propose a novel metric, *fast_p*, quantifying performance gain of generated kernels over PyTorch baselines *only when functionally correct*. Furthermore, we establish the first end-to-end GPU kernel generation evaluation framework, integrating LLM inference, iterative execution feedback, performance-profiler-driven optimization, and low-level operator verification. Experimental results show that state-of-the-art LLMs natively generate kernels achieving baseline performance on fewer than 20% of tasks. Incorporating execution feedback significantly improves success rates; however, gains diminish sharply as the required speedup threshold *p* increases.

To address CPU-induced communication bottlenecks in heterogeneous supercomputing, this paper systematically proposes and classifies the “GPU-centric communication” paradigm—fully migrating communication control from the CPU to the GPU to enable zero-CPU-intervention, end-to-end GPU-direct communication. Methodologically, it integrates GPUDirect RDMA, CUDA Graphs, programmable NICs (e.g., BlueField DPUs), and GPU kernel-level communication primitives to construct a GPU-native communication control stack. Key contributions include: (1) the first systematic characterization of hardware requirements—GPU memory coherence, NIC programmability, and kernel-driver support—and software challenges—including synchronization, error handling, and programming model adaptation; and (2) a design blueprint for low-latency, high-throughput, ultra-low-CPU-overhead communication infrastructure tailored for HPC and distributed AI training. (149 words)

This work addresses the challenge of optimizing and analyzing GPU kernels for deep convolution in cloud environments lacking hardware performance counters. The authors propose a counter-free performance analysis framework that integrates execution path decomposition with memory traffic modeling to systematically optimize the forward, input gradient, and weight gradient computation paths. Leveraging CUDA event timing, effective bandwidth estimation, and the roofline model, the approach employs optimization strategies such as shared memory tiling and warp-level partitioning. The resulting warp-tiled kernels achieve a 3.26× speedup over naive implementations, yielding a 1.29× end-to-end training acceleration. This significantly enhances the efficiency of critical computational paths and provides reproducible, architecture-level performance insights for resource-constrained cloud settings.

This study systematically investigates and quantifies the critical impact of CPU bottlenecks on large language model (LLM) inference performance in multi-GPU settings, where insufficient CPU resources often lead to suboptimal GPU utilization and elevated latency—even when GPUs are not fully saturated. Through comprehensive profiling of real-world LLM serving workloads, the work identifies key CPU-induced inefficiencies, including kernel launch delays, communication stalls, and tokenization overhead. Experimental results demonstrate that moderately increasing CPU core count—without adding more GPUs—can reduce time-to-first-token latency by 1.36× to 5.40× while substantially improving system stability and responsiveness. These findings underscore the necessity of balanced CPU-GPU provisioning for efficient LLM inference deployment.

This work addresses the inefficiency of forward and backward propagation in shallow neural networks on GPUs, which stems from memory access bottlenecks. To mitigate this issue, the authors propose a three-tier CUDA-based optimization strategy that integrates tiled shared-memory padding, pre-transposition of weight matrices, and fusion of matrix multiplication (MatMul) with ReLU activation operators. By enhancing memory access patterns, minimizing global memory round trips, and increasing computational density, the approach achieves a 1.41× speedup on an NVIDIA Tesla T4 GPU—reducing execution time from 21.0 seconds to 14.8 seconds. The optimized implementation significantly outperforms both serial CPU and OpenMP-parallelized baselines, demonstrating the effectiveness of the proposed techniques in accelerating training for shallow neural networks.

This work addresses the challenge of automatically generating high-performance GPU tiled kernels from high-level tensor algebra expressions, thereby alleviating the burden of manual optimization. The authors propose an end-to-end compilation framework that integrates layer-wise lowering, expression rewriting, automated schedule search, reduction fusion, and tiling optimizations. For the first time, this framework automatically discovers high-efficiency kernels—comparable to FlashAttention-3—from the mathematical specification of attention operators, while introducing a novel scheduler that preserves program structural regularity. Evaluated on GH200 and RTX 5090 GPUs, the generated kernels achieve up to 23% and 42% higher throughput, respectively, and match or surpass hand-optimized cuDNN kernels across multiple long-sequence configurations.

This work addresses the performance bottleneck in modern deep learning caused by the overhead of frequent GPU kernel launches due to numerous small-scale tensor operations. To mitigate this, the authors propose a persistent GPU kernel runtime system that continuously schedules computational tasks via a host-side task queue and dynamically loads operators through NVRTC-based just-in-time compilation and function pointer injection, thereby eliminating redundant kernel launches. The design introduces an innovative dual-slot aliasing mechanism for concurrent updates and a generic tensor abstraction, enabling transparent integration with PyTorch via TorchDispatch. Evaluated on representative workloads dominated by small operations, the system achieves up to 15.3× speedup over standard PyTorch, substantially improves GPU utilization, and maintains strong ecosystem compatibility.