ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

2026-05-02

AI Summary

This work addresses the inefficiency of existing Mixture-of-Experts (MoE) inference for discriminative tasks that require only the prefill phase, where synchronous expert parallelism incurs redundant computation and communication overhead. To overcome this limitation, the authors propose Asynchronous Expert Parallelism (AsyncEP), which decouples expert placement from activation routing. AsyncEP enables background streaming of expert weights and employs prefix-aware routing combined with real FLOPs-based load tracking for efficient scheduling. Furthermore, it eliminates synchronization bottlenecks by overlapping asynchronous AllGather communication with computation. Evaluated on Qwen3-235B-A22B, the approach achieves a 1.35–1.59× improvement in system throughput and attains GPU model FLOPs utilization of 29.8%–36.2%.

Abstract

Production LLM workloads increasingly serve discriminative tasks, such as classification, recommendation, and verification, whose answers are read from the logits of a single prefill pass with no autoregressive decoding. Serving these prefill-only workloads on mixture-of-experts (MoE) models is bottlenecked not by compute but by the distributed execution required to fit the model: existing parallel strategies (tensor, expert, and pipeline parallelism) trade memory pressure for redundant computation, communication, and synchronization, severely degrading MoE prefill serving efficiency. We observe that these overheads stem from coupling expert placement with synchronous activation routing -- a design inherited from the decoding era. The long, compute-bound forward passes of large-batch prefill open a per-layer window wide enough to stream expert weights in the background, replacing per-layer activation AllToAll with asynchronous weight AllGather fully overlapped with computation. We propose ZeRO-Prefill, a prefill-only serving system whose backend, AsyncEP (Asynchronous Expert Parallelism), gathers experts by weight rather than routing them by activation, and whose frontend co-enforces a physically-derived saturation threshold through prefix-aware routing and true-FLOPs load tracking. On Qwen3-235B-A22B across four hardware/precision configurations, ZeRO-Prefill delivers 1.35-1.37x throughput over the strongest distributed baseline on real-world workloads and up to 1.59x on long-context synthetic workloads, sustaining 29.8-36.2% per-GPU model FLOPs utilization.

Problem

• Mixture-of-Experts
• Prefill Serving
• Distributed Overhead
• Activation Routing
• Memory Bottleneck

Innovation

• ZeRO-Prefill
• Mixture-of-Experts
• Asynchronous Expert Parallelism
• Prefill-only Serving
• AllGather