TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes

2026-08-13

AI Summary

This work addresses the synchronization bottleneck in Expert Parallel (EP) Mixture-of-Experts (MoE) inference, where layer-wise synchronization is constrained by the slowest GPU. Existing load-balancing strategies fail because they overlook the nonlinear variation of expert execution times across memory- and compute-intensive regimes. The paper is the first to characterize this bimodal behavior and proposes a makespan-aware scheduling approach. It models expert execution using a max-affine time model coupled with a phase-diagram prediction mechanism, formulates per-batch scheduling as a fixed-cost makespan minimization problem, and designs an efficient solver for real-time adaptive scheduling. Experiments demonstrate up to 15.5% higher throughput under mixed workloads, 4–6% end-to-end throughput improvement on Qwen3-235B, and approximately 15.6% reduction in p99 latency, with the phase diagram accurately predicting deployment outcomes.

Abstract

In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU. Dispatchers balance token counts (EPLB, LPLB, UltraEP) or activated-expert counts (METRO), assuming expert time is linear in one. Measurements on two datacenter GPU generations show it is neither: below $\nstar\!\approx\!156$--$168$ tokens, HBM weight streaming dominates---cost attaches to \emph{activated replicas}, not tokens; above it, grouped GEMM rounds tokens to 128-tile $M$-tiles, so \emph{splitting} an expert adds padded compute. A max-affine profile $t=\max(a+bG,\,c+βN)$ captures both regimes. Realistic decode batches hold hot experts in the linear regime and cold in the flat \emph{simultaneously}; recorded batches show proxy dispatches differ by $1.4$--$1.6\times$ in modeled block time (p95 up to $1.7\times$), and \emph{which} proxy wins flips with the regime. We formalize per-batch dispatch as a fixed-charge makespan problem---NP-hard on two fully replicated GPUs, polynomial in degenerate limits---and present \sys{}, a makespan-aware dispatcher solving it in milliseconds off the critical path; its SGLang integration runs out-of-process and fuses dispatch with count collection into one in-graph kernel. Anchored by an 8-GPU Testbed~A microbenchmark, \sys{} stays within 1\% of the best fixed baseline everywhere and wins by up to $15.5\%$ where regimes mix. End-to-end on Testbed~B, Qwen3-235B (inside the win region) gains $4$--$6\%$ throughput and cuts p99 latency by ${\sim}15.6\%$; DeepSeek-V3 (outside, communication-dominated) shows only mechanism cost. A phase diagram, not a universal win, is the claim: it predicts both outcomes before deployment.

Problem

• expert-parallel
• load balancing
• makespan
• MoE serving
• nonlinear execution time

Innovation

• expert-parallel load balancing
• makespan-aware dispatching
• mixture-of-experts (MoE)
• phase-aware scheduling
• max-affine latency model