🤖 AI Summary
NVIDIA’s Multi-Instance GPU (MIG) lacks hardware-level power isolation, preventing accurate per-instance power attribution in multi-tenant cloud environments. Method: We propose the first online, fine-grained power modeling framework tailored for concurrent MIG workloads. We empirically demonstrate that conventional offline models fail significantly under MIG-shared workloads and introduce a lightweight, instance-level supervised learning framework driven by SM, CPU, and memory bandwidth utilization metrics. Contribution/Results: Evaluated on NVIDIA A100, our model achieves sub-3.2% mean absolute error across matrix multiplication and large language model inference workloads. It enables real-time, transparent carbon accounting and fair usage-based billing—establishing a new paradigm for granular GPU energy governance in cloud infrastructures.
📝 Abstract
Efficient power management in cloud data centers is essential for reducing costs, enhancing performance, and minimizing environmental impact. GPUs, critical for tasks like machine learning (ML) and GenAI, are major contributors to power consumption. NVIDIA's Multi-Instance GPU (MIG) technology improves GPU utilization by enabling isolated partitions with per-partition resource tracking, facilitating GPU sharing by multiple tenants. However, accurately apportioning GPU power consumption among MIG instances remains challenging due to a lack of hardware support. This paper addresses this challenge by developing software methods to estimate power usage per MIG partition. We analyze NVIDIA GPU utilization metrics and find that light-weight methods with good accuracy can be difficult to construct. We hence explore the use of ML-based power models to enable accurate, partition-level power estimation. Our findings reveal that a single generic offline power model or modeling method is not applicable across diverse workloads, especially with concurrent MIG usage, and that online models constructed using partition-level utilization metrics of workloads under execution can significantly improve accuracy. Using NVIDIA A100 GPUs, we demonstrate this approach for accurate partition-level power estimation for workloads including matrix multiplication and Large Language Model inference, contributing to transparent and fair carbon reporting.