ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill

2026-06-21

AI Summary

This work addresses the significant first-token latency and throughput limitations in online inference of Mixture-of-Experts (MoE) models, which stem from synchronous execution of attention and expert computations, causing imbalanced data-parallel workloads and severe synchronization bottlenecks. To overcome this, the paper proposes the first fully asynchronous MoE prefill pipeline that decouples the attention and expert computation stages, employs a disaggregated architecture, leverages asynchronous communication primitives, and co-optimizes request scheduling with model execution to eliminate global synchronization entirely. Evaluated on a CloudMatrix384 supersite, the proposed approach achieves a 90% higher SLO-compliant prefill throughput compared to the best existing synchronous baseline, substantially improving response latency and resource utilization.

Abstract

Mixture-of-Experts (MoE) models have become the de facto standard for scaling large language models. To maintain computational efficiency, modern MoE serving systems typically employ a hybrid parallelism strategy, combining Data Parallelism (DP) for attention stages with Expert Parallelism (EP) for MoE stages. However, this design necessitates frequent global synchronization barriers between attention DP groups and experts. In online serving, significant variance in request arrival rates and sequence lengths inherently leads to DP imbalance, causing severe synchronization stalls that degrade Time-to-First-Token (TTFT) and system throughput. We present ASAP, an asynchronous inference system specifically designed to accelerate the prefill phase of MoE models. ASAP disaggregates the attention and MoE stages and implements a fully asynchronous execution pipeline. This is achieved through a suite of specialized asynchronous communication primitives and four coordinated optimizations across request scheduling and model execution, which collectively dismantle global synchronization barriers. We implement and evaluate ASAP on CloudMatrix384 super-nodes, demonstrating that it improves SLO-compliant prefill throughput by 90% compared to state-of-the-art synchronous serving solutions.

Problem

• Mixture-of-Experts
• synchronization barrier
• prefill phase
• online serving
• throughput degradation

Innovation

• Mixture-of-Experts
• asynchronous inference
• disaggregated execution
• prefill acceleration
• synchronization elimination