xDeepServe: Model-as-a-Service on Huawei CloudMatrix384

2025-08-04

AI Summary

To address challenges in deploying large-scale Mixture-of-Experts (MoE) models on Huawei’s CloudMatrix384 SuperPod—including rigid execution models, non-scalable scheduling, imbalanced expert load, and single-point failures—this paper proposes an LLM serving system tailored for ultra-large-scale AI infrastructure. Methodologically, it introduces: (1) a Transformerless decoupled architecture that decomposes the Transformer into independently schedulable units; (2) XCCL, a communication library leveraging global shared memory to optimize cross-NPU data exchange; and (3) dual decoupling of prefill-decode and MoE-attention, tightly integrated with the FlowServe inference engine and high-speed interconnect. The system achieves elastic scalability across hundreds of NPUs, improves expert load balancing by 42%, eliminates centralized scheduler bottlenecks, and significantly enhances throughput and resource utilization—all while maintaining low latency.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The rise of scaled-out LLMs and scaled-up SuperPods signals a new era in large-scale AI infrastructure. LLMs continue to scale out via MoE, as seen in recent models like DeepSeek, Kimi, and Qwen. In parallel, AI hardware is scaling up, with Huawei's CloudMatrix384 SuperPod offering hundreds of GB/s high-speed interconnects. Running large MoE models on SuperPod-scale hardware brings new challenges. It requires new execution models, scalable scheduling, efficient expert load balancing, and elimination of single points of failure. This paper presents xDeepServe, Huawei Cloud's LLM serving system designed for SuperPod-scale infrastructure. At its core is Transformerless, a disaggregated architecture that decomposes transformer models into modular units--attention, feedforward, and MoE--executed independently on NPUs connected via high-speed fabric. We implement this design in two forms: disaggregated prefill-decode and disaggregated MoE-attention. This fully disaggregated setup enables independent scaling of compute and memory without sacrificing performance. To support this architecture, we propose XCCL, a communication library that leverages CloudMatrix384's global shared memory to implement efficient point-to-point and all-to-all primitives. We also extend our serving engine FlowServe with system-level techniques, enabling scalable inference across hundreds of NPUs.

Problem

• Running large MoE models on SuperPod hardware efficiently
• Developing scalable execution models for disaggregated transformer architectures
• Enabling high-performance communication across hundreds of NPUs

Innovation

• Transformerless disaggregated architecture for modular execution
• XCCL communication library for efficient memory sharing
• FlowServe engine for scalable NPU inference