Understanding the Synchronization Tax in GPU Scale-Up Domains

2026-08-23

AI Summary

研究通过分析GPU集群中的同步延迟问题，采用图算法和极端值理论揭示了GEMM内核执行时间的差异是主要因素，并提出该问题限制了带宽扩展的效益。

Abstract

GPU scale-up domains have become the building block of modern machine learning infrastructure, and their design follows a clear trajectory of exponential growth in both interconnect bandwidth and domain size. This paper argues that these two trends are in tension. Through a study of several hundred thousand collective operations across four language models and three recent GPU architectures, we find that GPUs within a scale-up domain arrive at collective barriers hundreds to thousands of microseconds apart, despite executing identical kernels on identical hardware over a uniform fabric. We call this waiting time the synchronization tax and show that it can consume over 50% of collective communication time in an 8-GPU scale-up domain. To understand the sources of this tax, we design a graph-based algorithm that operates on per-rank kernel traces, revealing that cross-rank variation in GEMM kernel execution times accounts for 78% of this overhead. We apply extreme value theory to model this variation and demonstrate that the synchronization tax grows with domain size. Folding this model into an augmented Hockney communication cost model, we show that the synchronization tax fundamentally limits the return on bandwidth scaling and inverts prevailing beliefs about how interconnect bandwidth should scale with domain size.

Problem

• Synchronization Tax
• GPU Scale-Up
• Collective Operations
• Interconnect Bandwidth
• Domain Size

Innovation

• synchronization tax
• cross-rank variation
• extreme value theory
• Hockney communication cost model