🤖 AI Summary
In distributed training, gradient communication overhead escalates with model and data scale, becoming a critical bottleneck; existing quantization schemes often lack AllReduce compatibility and theoretical convergence guarantees. This paper proposes Global-QSGD—the first family of globally scalar quantization operators that simultaneously ensures AllReduce compatibility, unbiasedness, and rigorous convergence guarantees. Unlike prior methods, Global-QSGD requires no error feedback and achieves a compression ratio improvement of $O(sqrt{n})$ over QSGD, while extending the theoretical framework for unbiased compression operators. Under standard assumptions, we prove that Global-QSGD preserves the convergence of distributed SGD. Experiments across diverse hardware platforms—including NVLink, PCIe, and cloud environments—demonstrate substantial reductions in communication volume, significant improvements in training throughput, and no degradation in model accuracy.
📝 Abstract
Efficient distributed training is a principal driver of recent advances in deep learning. However, communication often proves costly and becomes the primary bottleneck in these systems. As a result, there is a demand for the design of efficient communication mechanisms that can empirically boost throughput while providing theoretical guarantees. In this work, we introduce Global-QSGD, a novel family of quantization operators, engineered to accelerate distributed training based on global scaling. We demonstrate that Global-QSGD is the first theoretically rigorous Allreduce-compatible compression mechanism that achieves a provable speed-up by striking a balance between compression error and communication savings. Importantly, Global-QSGD does not rely on costly error feedback due to its inherent unbiasedness and offers up to $O(sqrt{n})$ additional compression ratio compared to the popular QSGD quantization ($n$ represents the number of workers). To obtain theoretical guarantees, we generalize the notion of standard unbiased compression operators to incorporate Global-QSGD. We show that this wider class permits standard analysis for unbiased compressors and thus ensures convergence for popular optimization algorithms (e.g., distributed SGD) under typical settings. For the empirical component of our work, we carry out a performance modeling analysis to determine if Global-QSGD can enhance training throughput under specific hardware configurations. We also conduct extensive empirical evaluations on various tasks, testing our theory on both NVLink and PCIe connections as well as a large-scale cloud system.
