nvidia nsight

Performance profiles of GPU kernels generated by tools such as Nsight Compute are rich in detail but are often challenging to interpret. To achieve the best performance possible on a given GPU architecture, kernel developers need to spend significant time analyzing and comparing profiles in the tool's graphical interface to identify and understand kernel performance bottlenecks. Large Language Models (LLMs) have shown promise in understanding complex data and generating natural language explanations. In this paper, we propose the Kernel Execution Explanation Toolkit (KEET), an LLM-based agentic framework for interpreting Nsight Compute profiles to generate useful and data-grounded natural language explanations of performance issues in GPU kernels, and suggestions for optimizations. We evaluate \toolname using several CUDA kernels of varying complexity on NVIDIA H100 GPUs. We find that the generated explanations, when provided as context, improve the quality of LLM code optimization and multiple-choice question answering in downstream tasks. We further demonstrate that the tool can be used to interpret performance data from large sets of profiles to improve the quality of optimization suggestions.

This work addresses the lack of transparency in NVIDIA’s closed-source user-space driver, which obscures the translation of CUDA API calls into hardware commands and impedes understanding of GPU behavior and performance attribution. The authors propose a novel approach that requires no modification to the proprietary driver, instead leveraging an open-source kernel driver, memory-mapped path instrumentation, and hardware watchpoints on the GPU’s doorbell registers to capture and reconstruct the complete low-level command stream with unprecedented accuracy. This methodology reveals the true DMA patterns and performance characteristics of CUDA data transfers and demonstrates that the low overhead of CUDA Graphs stems from their streamlined and efficient command submission mechanism. By significantly enhancing the interpretability of GPU runtime behavior, this approach establishes a new paradigm for middleware analysis and hardware-software co-design.

This work addresses three key challenges in real-time GPU performance monitoring at planetary scale: user privacy leakage, runtime overhead, and difficulty in data attribution across massive deployments. We propose the first end-to-end privacy-preserving GPU performance profiling architecture that operates at planetary scale with zero performance overhead. Methodologically, we design a lightweight kernel-level monitoring agent—built upon NSYS/NCU—that integrates differential privacy, secure aggregation, and distributed sampling to enable application-agnostic, anonymized kernel-level behavior attribution and resource accounting. Evaluated on a simulated 100,000-GPU cluster, our system achieves complete telemetry collection while enabling precise application-level attribution for realistic deep learning workloads (e.g., TorchBench) with no third-party privacy leakage. The architecture delivers trustworthy, scalable hardware performance insights for chip design and systems optimization.

Fine-grained, cross-platform real-time monitoring of GPU resources—particularly GPU memory peak usage and computational utilization—remains unsupported in Unix/Linux environments. Method: This paper introduces the first lightweight, dependency-free Python tool leveraging the NVIDIA Management Library (NVML) API. It employs multithreading and process-hooking techniques to enable low-overhead (average 0.3%) background sampling and precise peak capture of CPU/GPU utilization and system/GPU memory consumption. Contribution/Results: The tool unifies analysis across desktop and HPC environments with high accuracy (GPU memory peak error <2%). It enables job-level GPU resource profiling—the first such capability for fine-grained, runtime GPU characterization in HPC settings—thereby addressing a critical gap in production-grade GPU observability. The implementation is open-source and has been integrated into multiple scientific computing pipelines.

This study addresses the lack of native GPU monitoring tools in R for computational biology, where users traditionally rely on external command-line utilities. To bridge this gap, we introduce CudaMon, an R package that leverages the NVIDIA Management Library (NVML) interface to provide real-time monitoring of GPU utilization, memory usage, temperature, and power consumption. CudaMon seamlessly integrates data export and visualization capabilities into standard R workflows, significantly enhancing the efficiency and reproducibility of GPU-accelerated analyses. In a large-scale single-cell RNA-seq analysis involving millions of brain cells, the tool successfully identified stages—such as PCA and UMAP—where GPU utilization exceeded 90%, while also uncovering performance bottlenecks during data management phases.

This work addresses the challenge of optimizing and analyzing GPU kernels for deep convolution in cloud environments lacking hardware performance counters. The authors propose a counter-free performance analysis framework that integrates execution path decomposition with memory traffic modeling to systematically optimize the forward, input gradient, and weight gradient computation paths. Leveraging CUDA event timing, effective bandwidth estimation, and the roofline model, the approach employs optimization strategies such as shared memory tiling and warp-level partitioning. The resulting warp-tiled kernels achieve a 3.26× speedup over naive implementations, yielding a 1.29× end-to-end training acceleration. This significantly enhances the efficiency of critical computational paths and provides reproducible, architecture-level performance insights for resource-constrained cloud settings.

This study investigates the key performance bottlenecks limiting real-time GPU path tracing efficiency by comparing forward path tracing (PT) and wavefront path tracing (WPT). The former employs a megakernel design where a single thread processes an entire path, while the latter decomposes the algorithm into multiple specialized kernel stages that collaboratively maintain state buffers. Both approaches are implemented using NVIDIA RT Cores, and their performance is analyzed with Nsight Graphics. Experimental results demonstrate that WPT achieves approximately 16% higher performance than PT due to improved cache locality. Furthermore, profiling reveals that GPU compute units remain underutilized, indicating that memory latency and synchronization overhead constitute the primary bottlenecks in current real-time path tracing workloads.

This work addresses a critical limitation in evaluating large language models (LLMs) for CUDA program debugging: existing approaches often allow LLMs to bypass genuine bugs via performance-degrading fixes that merely satisfy test cases, thereby overestimating true debugging capability. To remedy this, we introduce CUDABeaver—the first benchmark specifically designed for performance-preserving CUDA debugging—comprising 213 tasks derived from real-world LLM failure cases. Each task enforces single-file editing constraints and provides error logs and build commands. We further propose a protocol-aware evaluation metric, pass@k(M,C,A), enabling multidimensional assessment. Experiments reveal that slightly tightening performance tolerance thresholds reduces the debugging success rate of seven state-of-the-art LLMs by up to 40 percentage points, exposing substantial optimistic bias in current evaluation practices.

This work proposes KernelPro, a closed-loop multi-agent system designed to automatically generate high-performance and energy-efficient GPU kernel code. By integrating large language models with hardware micro-benchmarking tools, KernelPro employs semantic feedback operators, a two-tier tool-calling architecture, a domain-adapted Monte Carlo Tree Search (MCTS) strategy, and direct CuTe source-code generation to jointly optimize for both performance and energy efficiency. Evaluated on KernelBench, KernelPro achieves up to a 5.30× speedup over baseline implementations. Furthermore, when applied to expert-optimized Mixture-of-Experts (MoE) kernels, it outperforms hand-tuned Triton kernels by 1.23× in performance while reducing measured energy consumption by 11.6%.