The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise

2026-04-10

AI Summary

This study investigates the true origins of expert specialization in Mixture-of-Experts (MoE) models, challenging the assumption that routing mechanisms reflect genuine domain-specific expertise. Through theoretical analysis and empirical evaluation, the work demonstrates for the first time that expert usage similarity is entirely determined by the geometric structure of hidden states rather than architecture-induced specialization. It further reveals that load-balancing losses suppress directions corresponding to shared representations, leading to a phenomenon termed “specialization collapse.” Additionally, routing patterns in pretrained MoEs prove largely semantically uninterpretable. Combining linear mapping analysis, geometric modeling of hidden states, cross-model and cross-layer routing comparisons, and theoretical derivation of loss functions, the study validates across five pretrained models a strong alignment between routing behavior and representation-space geometry: expert activations exhibit high overlap even on semantically unrelated inputs, and prompt-level routing fails to predict expert selection during actual inference.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Model-Based Reasoning
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their"expert specialization"remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, and specialization is therefore an emergent property of the representation space, not of the routing architecture itself. We confirm this at both token and sequence level across five pre-trained models. We additionally prove that load-balancing loss suppresses shared hidden state directions to maintain routing diversity, which might provide a theoretical explanation for specialization collapse under less diverse data, e.g. small batch. Despite this clean mechanistic account, we find that specialization patterns in pre-trained MoEs resist human interpretation: expert overlap between different models answering the same question is no higher than between entirely different questions ($\sim$60\%); prompt-level routing does not predict rollout-level routing; and deeper layers exhibit near-identical expert activation across semantically unrelated inputs, especially in reasoning models. We conclude that, while the efficiency perspective of MoEs is well understood, understanding expert specialization is at least as hard as understanding LLM hidden state geometry, a long-standing open problem in the literature.

Problem

• Mixture of Experts
• expert specialization
• routing mechanism
• hidden state geometry
• representation space

Innovation

• Mixture of Experts
• expert specialization
• hidden state geometry
• routing mechanism
• load-balancing loss