Evaluating Expert Contributions in a MoE LLM for Quiz-Based Tasks

2025-02-24

AI Summary

This work addresses the underexplored problem of expert contribution assessment in Mixture-of-Experts (MoE) large language models under the MMLU question-answering benchmark. We systematically analyze expert activation patterns, gating distribution properties, and intra-layer expert performance disparities during inference. By empirically examining gating outputs and expert accuracies across MMLU subtasks, we make three key findings: (1) a substantial fraction of experts remain entirely inactive during inference; (2) gating logits exhibit high uniformity—contradicting the canonical sparse activation assumption; and (3) average accuracy differences among experts within the same MoE layer exceed 20%, revealing severe capability heterogeneity. These results challenge two foundational assumptions underlying MoE architectures—namely, “sparse efficiency” and “expert homogeneity”—and provide critical empirical evidence for model diagnostics, gating mechanism redesign, and expert collaboration optimization.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recently, Large Language Models (LLMs) with Mixture of Experts (MoE) layers have gained significant attention. Currently, state-of-the-art LLMs utilize this architecture. There is a substantial amount of research on how to train such models and how to select hyperparameters for this architecture. However, there is a lack of studies focusing on post-evaluation analysis of MoE layer properties. In this paper, we take a first step toward closing this gap by evaluating expert contributions on the quiz-based MMLU benchmark. We show that most experts were never activated during inference on this benchmark. Additionally, the output distribution of gating networks is much closer to uniform than sparse. Finally, we demonstrate that the average performance of some experts within the same layer varies significantly.

Problem

• Evaluate expert contributions in MoE LLMs
• Analyze activation patterns in MoE layers
• Assess performance variability among experts

Innovation

• Evaluates MoE expert contributions
• Focuses on MMLU quiz benchmarks
• Analyzes gating network distributions