GeoMesh: Workload-Balanced and Sign-Compressed Geo-Distributed LLM Training

2026-09-16

AI Summary

为解决地理分布式LLM训练中的同步和异步方法问题，GeoMesh通过平衡工作负载和压缩通信量来提高效率和准确性。

Abstract

Large language models are increasingly trained on GPUs distributed across multiple regions, but geo-distributed training is challenging in practice. Real clusters often contain GPUs with different speeds and memory capacities, and they communicate over slow wide-area networks. Our analysis shows that this creates serious problems: existing synchronous methods preserve stable updates, but fast GPUs wait up to 20.9% of their runtime for slower ones, and all workers spend, on average, 65.8% of their runtime on synchronization. Recent asynchronous methods reduce waiting time but worsen the model accuracy due to stale updates. To address the problems, we present GeoMesh, a synchronous geo-distributed training framework for heterogeneous GPUs. GeoMesh balances per-worker workloads by assigning each GPU a suitable batch size and number of inner steps, so faster GPUs do more useful work instead of waiting. It also reduces communication volume by nearly 32x by exchanging compressed sign-based pseudo-gradients with lightweight magnitude and token count. Across heterogeneous GPUs and Azure-derived WAN, GeoMesh reduces time-to-target perplexity by up to 70.2% over representative baselines and lowers straggler- and WAN-induced GPU idle by up to 8.0x and 5.6x, respectively, while preserving comparable zero-shot accuracy.

Problem

• geo-distributed training
• heterogeneous GPUs
• wide-area networks
• synchronization
• stale updates

Innovation

• synchronous geo-distributed training
• workload balancing
• compressed sign-based pseudo-gradients
• heterogeneous GPUs