Bandwidth-Aware and Cost-Efficient Pipeline Parallel Scheduling in Geo-Distributed LLM Training

2026-05-24

AI Summary

This work addresses the challenges of training large language models across geographically distributed GPU clusters, where heterogeneous network bandwidth and regional electricity price disparities hinder existing pipeline parallelism strategies from simultaneously minimizing job completion time and power cost, while also suffering from head-of-line blocking. To overcome these limitations, the authors propose BACE-Pipe, a novel framework that jointly models real-time network utilization and job characteristics for the first time. BACE-Pipe integrates dynamic priority scheduling, bandwidth-aware path planning, and electricity-price-aware GPU allocation to co-optimize pipeline execution across regions. Experimental results demonstrate that the approach effectively mitigates head-of-line blocking and significantly reduces both latency and operational cost in multi-tenant environments, achieving 27.9%–64.7% lower average job completion time and 12.6%–30.6% reduction in total electricity expenditure.

Abstract

The rapid evolution of large language models (LLMs) has made geographically distributed training necessary due to GPU scarcity within a single cloud region. In such cross-region settings, Pipeline Parallelism (PP) is communication-efficient, yet scheduling PP remains challenging under heterogeneous inter-region bandwidth and regional electricity prices. Existing schedulers are either delay-first, incurring high electricity cost, or cost-first, relying on rigid resource allocation that prolongs Job Completion Time (JCT). They are also ineffective at optimizing execution order in multi-tenant environments, where long-running and bandwidth-intensive jobs can cause head-of-line (HoL) blocking and degrade overall performance. To this end, we propose BACE-Pipe, a bandwidth-aware and cost-efficient pipeline scheduling framework for LLM training across geo-distributed clusters. BACE-Pipe first introduces a dynamic job prioritization mechanism that optimizes execution order by jointly considering job characteristics (e.g., computation time) and real-time network utilization. It then employs a bandwidth-aware pathfinder to identify feasible cross-region pipeline paths that satisfy communication constraints, thereby preventing communication from stalling the pipeline. Among all feasible paths, a cost-minimizing allocator determines the optimal GPU placement strategy by preferentially assigning resources to regions with lower electricity prices. Consequently, BACE-Pipe mitigates HoL blocking, improves resource utilization, and simultaneously reduces both JCT and total electricity cost. Extensive simulations show that BACE-Pipe reduces average JCT by 27.9%--64.7% and total electricity cost by 12.6%--30.6% compared with state-of-the-art baselines.

Problem

• Pipeline Parallelism
• Geo-Distributed Training
• Bandwidth Heterogeneity
• Electricity Cost
• Head-of-Line Blocking

Innovation

• pipeline parallelism
• geo-distributed training
• bandwidth-aware scheduling
• cost-efficient allocation
• head-of-line blocking