Empirical Analysis of GPU Frequency Behavior Under ML Workloads

2026-07-09

AI Summary

This study addresses a critical limitation in existing machine learning latency prediction methods, which typically assume GPU kernel execution times are independent and overlook inter-kernel timing dependencies introduced by dynamic frequency scaling. Through empirical characterization of NVIDIA GPU frequency behavior under ML/AI workloads, we systematically reveal—for the first time—that recent workload history within an approximately 80 ms window significantly influences operating frequency, particularly on lower-end GPUs. Leveraging GPU performance monitoring, kernel-level latency measurements, and frequency trajectory analysis, we demonstrate that conventional latency models based on summing independent kernel durations exhibit substantial inaccuracies. These findings challenge a foundational assumption in mainstream latency modeling and provide crucial insights for designing novel scheduling and optimization mechanisms that jointly account for frequency dynamics, latency, and energy efficiency.

Abstract

This work presents ongoing research on the frequency scaling behavior of NVIDIA GPUs when executing ML/AI workloads. Our preliminary findings show that, on lower-performance GPUs, the operating frequency is strongly affected by the recent workload history, typically within an 80ms window. This behavior challenges a common assumption underlying several state-of-the-art ML latency-prediction techniques, which treat individual GPU kernel latencies as independent and therefore estimate total execution time by summing isolated per-kernel measurements. Our results indicate that such an assumption does not always hold, as the GPU's dynamic frequency scaling introduces inter-kernel dependencies. We also outline several promising directions for leveraging this observation in future work, including improved latency-prediction models, GPU kernel-reordering strategies, and NAS-driven guidelines for frequency/latency/energy-aware model design.

Problem

• GPU frequency scaling
• ML latency prediction
• kernel dependency
• dynamic frequency
• workload history

Innovation

• GPU frequency scaling
• ML workload dependency
• kernel latency prediction
• dynamic voltage and frequency scaling (DVFS)
• neural architecture search (NAS)