Measuring GPU utilization one level deeper

2025-01-28

AI Summary

To address the coexistence of performance instability and resource underutilization in multi-application GPU co-location, this paper introduces the first kernel-level, fine-grained resource interference quantification framework spanning multiple hardware layers—including compute units, L1/L2 caches, and memory bandwidth—overcoming the limitations of conventional coarse-grained utilization-based modeling. Leveraging micro-benchmarks, hardware performance counter sampling, kernel-level isolation experiments, and interference modeling, the framework enables reproducible characterization of interference behavior across critical subsystems. Based on this, we design a dynamic co-location scheduler with strict service-level objective (SLO) guarantees, achieving over 35% improvement in aggregate GPU utilization while maintaining quality-of-service requirements. This work establishes both theoretical foundations and empirical validation for predictable, high-performance GPU resource sharing.

Technology Category

• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

GPU hardware is vastly underutilized. Even resource-intensive AI applications have diverse resource profiles that often leave parts of GPUs idle. While colocating applications can improve utilization, current spatial sharing systems lack performance guarantees. Providing predictable performance guarantees requires a deep understanding of how applications contend for shared GPU resources such as block schedulers, compute units, L1/L2 caches, and memory bandwidth. We propose a methodology to profile resource interference of GPU kernels across these dimensions and discuss how to build GPU schedulers that provide strict performance guarantees while colocating applications to minimize cost.

Problem

• Multi-application
• GPU resource management
• Performance stabilization

Innovation

• GPU resource management
• performance stability
• cost efficiency