A Survey on Inference Optimization Techniques for Mixture of Experts Models

2024-12-18

AI Summary

To address cross-layer optimization challenges—including low inference speed, high energy consumption, and poor hardware utilization—in large-scale Mixture-of-Experts (MoE) model deployment, this paper proposes the first unified taxonomy spanning model, system, and hardware stacks. Methodologically, it integrates expert pruning, dynamic routing with load balancing, multi-granularity compression (pruning, quantization, distillation), distributed scheduling, and hardware-aware compilation. We innovatively establish a cross-layer co-optimization framework and open-source an actively maintained *Awesome MoE Inference* knowledge repository. Our contributions include a structured technical landscape that clarifies core challenges and evolutionary trends, significantly improving inference efficiency and energy efficiency—enabling low-latency, high-throughput, and power-efficient industrial MoE deployment.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

The emergence of large-scale Mixture of Experts (MoE) models represents a significant advancement in artificial intelligence, offering enhanced model capacity and computational efficiency through conditional computation. However, deploying and running inference on these models presents significant challenges in computational resources, latency, and energy efficiency. This comprehensive survey analyzes optimization techniques for MoE models across the entire system stack. We first establish a taxonomical framework that categorizes optimization approaches into model-level, system-level, and hardware-level optimizations. At the model level, we examine architectural innovations including efficient expert design, attention mechanisms, various compression techniques such as pruning, quantization, and knowledge distillation, as well as algorithm improvement including dynamic routing strategies and expert merging methods. At the system level, we investigate distributed computing approaches, load balancing mechanisms, and efficient scheduling algorithms that enable scalable deployment. Furthermore, we delve into hardware-specific optimizations and co-design strategies that maximize throughput and energy efficiency. This survey provides both a structured overview of existing solutions and identifies key challenges and promising research directions in MoE inference optimization. To facilitate ongoing updates and the sharing of cutting-edge advances in MoE inference optimization research, we have established a repository accessible at https://github.com/MoE-Inf/awesome-moe-inference/.

Problem

• Massive Mixture of Experts (MoE) Models
• Performance Optimization
• Hardware Utilization

Innovation

• MoE Model Optimization
• Load Balancing and Scheduling
• Hardware Efficiency Enhancement