dcgm

This work addresses the low efficiency of GPU resource and power utilization in heterogeneous high-performance computing (HPC) systems, as well as the lack of accurate predictive methods. To tackle these challenges, the authors propose a two-stage prediction framework that uniquely integrates Slurm job logs with fine-grained NVIDIA Data Center GPU Manager (DCGM) metrics. Leveraging only job submission features, the framework achieves high-accuracy predictions of application-level average power consumption, peak GPU utilization, and memory utilization. Experimental results demonstrate prediction accuracies of 97% for peak GPU utilization and 92% for runtime power consumption, significantly enhancing scheduling efficiency and power management capabilities in HPC environments. The study also validates the effectiveness of DCGM metrics in characterizing application behavior.

Fine-grained, cross-platform real-time monitoring of GPU resources—particularly GPU memory peak usage and computational utilization—remains unsupported in Unix/Linux environments. Method: This paper introduces the first lightweight, dependency-free Python tool leveraging the NVIDIA Management Library (NVML) API. It employs multithreading and process-hooking techniques to enable low-overhead (average 0.3%) background sampling and precise peak capture of CPU/GPU utilization and system/GPU memory consumption. Contribution/Results: The tool unifies analysis across desktop and HPC environments with high accuracy (GPU memory peak error <2%). It enables job-level GPU resource profiling—the first such capability for fine-grained, runtime GPU characterization in HPC settings—thereby addressing a critical gap in production-grade GPU observability. The implementation is open-source and has been integrated into multiple scientific computing pipelines.

This study addresses the lack of native GPU monitoring tools in R for computational biology, where users traditionally rely on external command-line utilities. To bridge this gap, we introduce CudaMon, an R package that leverages the NVIDIA Management Library (NVML) interface to provide real-time monitoring of GPU utilization, memory usage, temperature, and power consumption. CudaMon seamlessly integrates data export and visualization capabilities into standard R workflows, significantly enhancing the efficiency and reproducibility of GPU-accelerated analyses. In a large-scale single-cell RNA-seq analysis involving millions of brain cells, the tool successfully identified stages—such as PCA and UMAP—where GPU utilization exceeded 90%, while also uncovering performance bottlenecks during data management phases.

NVIDIA’s Multi-Instance GPU (MIG) lacks hardware-level power isolation, preventing accurate per-instance power attribution in multi-tenant cloud environments. Method: We propose the first online, fine-grained power modeling framework tailored for concurrent MIG workloads. We empirically demonstrate that conventional offline models fail significantly under MIG-shared workloads and introduce a lightweight, instance-level supervised learning framework driven by SM, CPU, and memory bandwidth utilization metrics. Contribution/Results: Evaluated on NVIDIA A100, our model achieves sub-3.2% mean absolute error across matrix multiplication and large language model inference workloads. It enables real-time, transparent carbon accounting and fair usage-based billing—establishing a new paradigm for granular GPU energy governance in cloud infrastructures.

This work addresses the high development and deployment costs incurred by maintaining multiple firmware images for GPU servers due to minor hardware variations. To resolve this, the authors propose a unified firmware approach based on runtime hardware identification: during system boot, hardware identity is queried via D-Bus, and a corresponding declarative JSON configuration is dynamically loaded and injected into downstream services as environment variables. This design fully abstracts platform-specific differences into configuration files, eliminating the need for multiple firmware images. Experimental results demonstrate that a single firmware image can efficiently support diverse hardware variants, substantially reducing maintenance complexity and improving deployment efficiency. The core innovation lies in a lightweight, JSON-driven platform abstraction mechanism that integrates seamlessly with the OpenBMC architecture.

This work addresses the lack of transparency in NVIDIA’s closed-source user-space driver, which obscures the translation of CUDA API calls into hardware commands and impedes understanding of GPU behavior and performance attribution. The authors propose a novel approach that requires no modification to the proprietary driver, instead leveraging an open-source kernel driver, memory-mapped path instrumentation, and hardware watchpoints on the GPU’s doorbell registers to capture and reconstruct the complete low-level command stream with unprecedented accuracy. This methodology reveals the true DMA patterns and performance characteristics of CUDA data transfers and demonstrates that the low overhead of CUDA Graphs stems from their streamlined and efficient command submission mechanism. By significantly enhancing the interpretability of GPU runtime behavior, this approach establishes a new paradigm for middleware analysis and hardware-software co-design.

This work addresses the lack of efficient and intuitive visualization tools for monitoring GPU-accelerated resources in current supercomputing systems. The authors propose a novel approach that integrates a 3D game engine with multi-source performance data streams to construct a three-dimensional digital twin of a supercomputer within the TX-Digital Twin platform. For the first time, critical GPU metrics—including memory utilization, temperature, and power consumption—are embedded into an immersive 3D visualization environment. Through optimized rendering calls, the system achieves real-time and historical visualization of GPU performance indicators with high clarity and minimal overhead. This significantly enhances the intuitiveness and operability of supercomputing resource monitoring, offering system administrators and users an effective means to observe and manage complex GPU workloads.

Existing GPU power estimation methods often suffer from low accuracy, limited flexibility, or outdated architectural assumptions, making them inadequate for fine-grained energy analysis in modern high-performance computing. To address this gap, this work proposes Wattchmen—a high-fidelity, cross-architecture, instruction-level GPU power modeling framework. By constructing instruction energy models calibrated with diverse microbenchmarks, Wattchmen enables accurate power prediction and attribution across architectures such as V100, A100, and H100 under varying cooling conditions. Experimental evaluation on 16 representative workloads demonstrates that Wattchmen achieves an average absolute percentage error as low as 14% on the V100, substantially outperforming AccelWattch and Guser. Furthermore, it successfully guided energy optimizations for Backprop and QMCPACK, yielding up to 35% energy savings.

This work addresses the challenge of accurately predicting inference power consumption and per-token latency for unseen large language model (LLM) and GPU combinations without empirical measurements, a key bottleneck in energy-efficient deployment. The authors propose WattGPU, the first framework to predict both metrics using only publicly available LLM metadata (e.g., parameter count, architecture) and NVIDIA datacenter GPU specifications, eliminating the need for hardware access or performance profiling. Employing leave-one-GPU/LLM-out cross-validation, WattGPU demonstrates strong generalization to unseen models and hardware, achieving median absolute percentage errors of just 13.5% for power and 8.5% for latency, along with GPU ranking Kendall τ coefficients of at least 0.76—representing a 2–4× error reduction over conventional baselines.

GPU nodes are central to modern HPC and AI workloads, yet many failures do not manifest as immediate hard faults. While some instabilities emerge gradually as weak thermal or efficiency drift, a significant class occurs abruptly with little or no numeric precursor. In these detachment-class failures, GPUs become unavailable at the driver or interconnect level and the dominant observable signal is structural, including disappearance of device metrics and degradation of monitoring payload integrity. This paper proposes an observability-aware early-warning framework that jointly models (i) utilization-aware thermal drift signatures in GPU telemetry and (ii) monitoring-pipeline degradation indicators such as scrape latency increase, sample loss, time-series gaps, and device-metric disappearance. The framework is evaluated on production telemetry from GPU nodes at GWDG, where GPU, node, monitoring, and scheduler signals can be correlated. Results show that detachment failures exhibit minimal numeric precursor and are primarily observable through structural telemetry collapse, while joint modeling increases early-warning lead time compared to GPU-only detection. The dataset used in this study is publicly available at https://doi.org/10.5281/zenodo.19052367.