UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods

2026-07-07

AI Summary

This work addresses the inefficiencies in deploying sparse Mixture-of-Experts (MoE) models on high-performance supercomputing clusters, where coarse-grained Bulk Synchronous Parallel (BSP) scheduling leads to serialization bottlenecks, excessive synchronization overhead, and load imbalance due to obliviousness to communication distance. To overcome these limitations, the authors propose UBEP, a unified bus-based expert parallel communication library that re-engineers the All-to-All primitive by integrating fine-grained asynchronous communication, distance-aware task scheduling, and lightweight synchronization mechanisms. Leveraging a unified global address space and high-bandwidth interconnect architecture, UBEP substantially enhances communication efficiency: experiments demonstrate up to a 52.4% reduction in All-to-All latency and an 11.1% decrease in time per output token (TPOT) during MoE inference.

Abstract

The deployment of Mixture-of-Experts (MoE) models on production high-bandwidth superpods, such as NVIDIA's NVL72/576 and Huawei's CloudMatrix384, introduces critical challenges beyond raw interconnect bandwidth. While these systems provide unified global address spaces and high-bandwidth fabrics, their full potential for sparse MoE communication is hindered by three fundamental bottlenecks: (1) Strict execution serialization imposed by coarse-grained Bulk Synchronous Parallel (BSP) orchestration of interdependent communication phases; (2) Prohibitive synchronization overhead that fails to scale alongside high interconnect bandwidth; and (3) Severe load imbalance resulting from distance-agnostic scheduling of irregular token traffic. To eliminate these bottlenecks, we introduce UBEP (Unified-Bus Expert Parallelism), a production-ready communication library that rethinks MoE's All-to-All primitives for modern superpod architectures. Through large scale experiments, UBEP reduces All-to-All latency by up to 52.4% and MoE inference Time Per Output Token (TPOT) by up to 11.1%.

Problem

• Mixture-of-Experts
• All-to-All communication
• superpod
• load imbalance
• synchronization overhead

Innovation

• Mixture-of-Experts
• Expert Parallelism
• All-to-All Communication
• Superpod
• Load Balancing