Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications

2025-07-03

AI Summary

This study systematically investigates the dual impact of computation-communication overlap on performance and energy efficiency in GPU-accelerated distributed deep learning. Addressing the challenges of increased computational latency, resource contention, and elevated power consumption under aggressive overlap, we build a multi-GPU distributed training testbed to evaluate overlap behavior across varying numerical precisions, specialized compute units (e.g., Tensor Cores), and power capping regimes. Results show that while overlap improves average throughput by 10.2%, it incurs up to 40% higher computational latency and an average 18.9% slowdown in computation; significant energy-efficiency trade-offs emerge under power and frequency constraints. We propose a novel “balanced overlap” strategy, demonstrating that hardware-specific characteristics—rather than overlap per se—are the primary determinants of its efficacy. Our findings provide empirical evidence and design principles for energy-aware distributed training systems.

Technology Category

• Machine Learning: Hardware-aware ML
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Distributed Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments

Abstract

This paper provides an in-depth characterization of GPU-accelerated systems, to understand the interplay between overlapping computation and communication which is commonly employed in distributed training settings. Due to the large size of models, distributing them across multiple devices is required. Overlapping strategies, which enable concurrent computation and communication, are critical for mitigating communication bottlenecks and maximizing GPU utilization. However, the current consensus is that we should always and aggressively overlap compute and communication to mitigate the overhead of distribution. By systematically evaluating state-of-the-art GPUs, this study investigates the impact of hardware features such as numeric precision, specialized cores, and power capping on distributed training workloads. Comprehensive experiments and studies showcase the effects of overlapping strategies on performance and power consumption across varying scenarios. We observe that overlapping computation and communication can result in an average computational slowdown of 18.9%, with a maximum of 40.0% slowdown. This slowdown is in comparison to the scenario when no communication was happening with the compute. We consider this an ideal execution scenario, where the communication in parallel has not impact on the compute time. However, performing computation and communication sequentially is, on average, 10.2% slower than overlapped execution, with a maximum slowdown of 26.6%. We further observe, while specialized datapath and optimized numeric precision mitigate certain slowdowns, overlapping execution can lead to resource contention and also increase power consumption under specific configurations. The analysis also uncovers trade-offs introduced by power and frequency capping, emphasizing the importance of balanced strategies to optimize energy efficiency and training throughput.

Problem

• Analyzes compute-communication overlap impact on GPU-accelerated distributed deep learning.
• Investigates hardware features' effect on performance and power in distributed training.
• Explores trade-offs between overlapping strategies, resource contention, and energy efficiency.

Innovation

• Overlapping computation and communication in GPUs
• Evaluating hardware impact on distributed training
• Balancing power capping and performance trade-offs