Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switch

2026-06-25

AI Summary

This work addresses the performance limitations of Mixture-of-Experts (MoE) models during inference, which arise from the static choice between tensor parallelism (TP) and expert parallelism (EP)—a rigidity that fails to adapt to dynamic request concurrency. To overcome this, the authors propose Moebius, the first system enabling zero-downtime, low-overhead runtime switching between TP and EP layouts. Moebius achieves efficient resharding of shared expert weights and KV caches through fused communication kernels, a fixed-address resharding mechanism, and dual-layout resident memory management. Evaluated on an 8×H200 cluster with Qwen3-235B-A22B, Moebius consistently matches the throughput of the optimal static strategy, accelerates reinforcement learning inference by 1.16–1.25×, incurs switching latencies of only 215–434 ms, and adds merely 2.4% memory overhead.

Abstract

Mixture-of-Experts (MoE) architectures scale large language models (LLMs) to hundreds of billions of parameters. Serving a single MoE model requires multiple GPUs operating in parallel, typically through tensor parallelism (TP) or expert parallelism (EP). The optimal choice depends on the number of in-flight requests: TP is faster at low concurrency, whereas EP wins at high concurrency. Production workloads cross this boundary continually: online serving sees bursty arrivals that subside into quiet periods, and reinforcement-learning rollouts begin as a high-concurrency burst that decays into a long tail of stragglers. Pinning either layout therefore forfeits performance when the workload crosses to the other side. We present Moebius, a serving system that switches between EP and TP at runtime without restarting the engine or dropping in-flight requests. Our key insight is that EP and TP are two layouts of one model, not two models: they compute the same function over byte-identical expert weights and KV cache, so a switch changes only which rank owns each slice. Moving those owner-changed slices is the sole irreducible cost, and modern high-bandwidth GPU interconnects make it fast enough to do between decode steps without draining in-flight requests. Moebius preserves each parallelism's runtime resident, and reshards the single copy of expert weights and KV cache at fixed addresses with fused GPU-to-GPU transfer kernels. On 8x H200 GPUs serving Qwen3-235B-A22B, Moebius matches the better static parallelism at every operating point, and beats it on RL rollouts by 1.16-1.25x across steps. Each switch completes in 215-434 ms, and Moebius holds both layouts resident with only 2.4% memory overhead.

Problem

• Mixture-of-Experts
• parallelism switching
• LLM serving
• tensor parallelism
• expert parallelism

Innovation

• Mixture-of-Experts
• Runtime Parallelism Switching
• Expert Parallelism
• Tensor Parallelism
• Model Serving