ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training

2026-05-22

AI Summary

This work addresses the challenges of low communication efficiency and system complexity in large-scale AI training across geographically distributed data centers. It presents the first systematic characterization and joint optimization of three critical dimensions in “scale-across” training: parallelism strategy deployment, job scheduling, and network transport. By co-designing parallel placement, scheduling policies, and advanced networking techniques—and validating the approach through both real-world testbeds and large-scale simulations—the study achieves end-to-end global optimization of computation and communication. Experimental results demonstrate that the proposed solution improves training throughput by up to 64.62% over current production configurations and by 37.59% compared to state-of-the-art baselines.

Abstract

The rapid scaling of large language model training requires distributing GPU resources across multiple data center buildings and regions. We refer to such paradigm as "scale-across" training. As infrastructure expands, the system design space becomes increasingly intricate, encompassing new model architectures, hardware heterogeneity, and evolving communication patterns. Drawing from Meta's production experience, we highlight the complexities of deploying training jobs across a few data centers housing hundreds of thousands of GPUs. To accelerate exploration of the large design space and to enable efficient training for frontier model development, we conduct in-depth characterization of three key design dimensions: parallelism placement, parallelism scheduling, and network layer technologies. We then propose ScaleAcross Explorer, an optimizer that considers the interplay of design dimensions and holistically optimizes scale-across training. Testbed experiments and simulations demonstrate up to 64.62% training speedups over production configuration and up to 37.59% training speedups over the state-of-the-art baseline across a wide range of design points.

Problem

• scale-across
• distributed training
• communication optimization
• large language models
• data center

Innovation

• scale-across training
• communication optimization
• parallelism placement
• parallelism scheduling
• network layer technologies