MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints

2025-04-12

AI Summary

To address low inference throughput of Mixture-of-Experts (MoE) large language models caused by GPU memory constraints, this paper proposes a full-stack CPU-GPU co-execution optimization framework. First, we develop a fine-grained performance model integrating hardware characteristics—CPU memory bandwidth, GPU compute capacity, and expert routing load—with system execution semantics to precisely identify bottlenecks and approach theoretical throughput limits. Second, we design a dynamic resource scheduling and computation offloading strategy supporting weight sharding, elastic expert loading, and cross-device pipelined execution. Evaluated across multiple MoE models and datasets, our approach achieves an average 4.6× throughput improvement (up to 25.5×), with performance prediction error under 6% (94% accuracy). This work is the first to enable hardware-aware, full-stack modeling and real-time collaborative scheduling for MoE inference, significantly alleviating the GPU memory wall bottleneck in high-throughput deployment.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Mixture of Experts (MoE) LLMs, characterized by their sparse activation patterns, offer a promising approach to scaling language models while avoiding proportionally increasing the inference cost. However, their large parameter sizes present deployment challenges in resource-constrained environments with limited GPU memory capacity, as GPU memory is often insufficient to accommodate the full set of model weights. Consequently, typical deployments rely on CPU-GPU hybrid execution: the GPU handles compute-intensive GEMM operations, while the CPU processes the relatively lightweight attention mechanism. This setup introduces a key challenge: how to effectively optimize resource utilization across CPU and GPU? Prior work has designed system optimizations based on performance models with limited scope. Specifically, such models do not capture the complex interactions between hardware properties and system execution mechanisms. Therefore, previous approaches neither identify nor achieve the hardware limit. This paper presents MoE-Lens, a high-throughput MoE LLM inference system designed through holistic performance modeling for resource-constrained environments. Our performance model thoroughly analyzes various fundamental system components, including CPU memory capacity, GPU compute power, and workload characteristics, to understand the theoretical performance upper bound of MoE inference. Furthermore, it captures the system execution mechanisms to identify the key hardware bottlenecks and accurately predict the achievable throughput. Informed by our performance model, MoE-Lens introduces an inference system approaching hardware limits. Evaluated on diverse MoE models and datasets, MoE-Lens outperforms the state-of-the-art solution by 4.6x on average (up to 25.5x), with our theoretical model predicting performance with an average 94% accuracy.

Problem

• Optimize CPU-GPU resource use for MoE LLM inference
• Achieve hardware performance limits in resource-constrained environments
• Model system interactions to predict throughput accurately

Innovation

• Holistic performance modeling for MoE LLM
• Optimizes CPU-GPU hybrid execution efficiently
• Achieves near-hardware-limit throughput performance