On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating

2025-05-16

AI Summary

This paper addresses the lack of statistical foundations for the shared-expert mechanism and normalized sigmoid gating in DeepSeekMoE. From a statistical learning perspective, it systematically analyzes their design advantages via convergence analysis and expert estimation modeling. Theoretically, it first proves that shared experts significantly improve sample efficiency; it further introduces and quantifies the statistical gain of normalized sigmoid gating, revealing intrinsic trade-offs among router saturation, sensitivity (derivative), and expert utilization. Empirical validation on synthetic data and multimodal (vision/language) benchmarks confirms consistent improvements in both task performance and load balancing. The core contribution is the establishment of the first unified statistical framework explaining both mechanisms, rigorously characterizing their dual benefits for sample efficiency and generalization.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Mixture of experts (MoE) methods are a key component in most large language model architectures, including the recent series of DeepSeek models. Compared to other MoE implementations, DeepSeekMoE stands out because of two unique features: the deployment of a shared expert strategy and of the normalized sigmoid gating mechanism. Despite the prominent role of DeepSeekMoE in the success of the DeepSeek series of models, there have been only a few attempts to justify theoretically the value of the shared expert strategy, while its normalized sigmoid gating has remained unexplored. To bridge this gap, we undertake a comprehensive theoretical study of these two features of DeepSeekMoE from a statistical perspective. We perform a convergence analysis of the expert estimation task to highlight the gains in sample efficiency for both the shared expert strategy and the normalized sigmoid gating, offering useful insights into the design of expert and gating structures. To verify empirically our theoretical findings, we carry out several experiments on both synthetic data and real-world datasets for (vision) language modeling tasks. Finally, we conduct an extensive empirical analysis of the router behaviors, ranging from router saturation, router change rate, to expert utilization.

Problem

• Theoretical justification for shared expert strategy in DeepSeekMoE
• Statistical analysis of normalized sigmoid gating mechanism
• Convergence and efficiency gains in expert estimation tasks

Innovation

• Shared expert strategy enhances sample efficiency
• Normalized sigmoid gating improves convergence
• Comprehensive theoretical and empirical analysis conducted