On the Spatial Structure of Mixture-of-Experts in Transformers

2025-04-06

AI Summary

Conventional MoE routing mechanisms assume expert selection relies solely on semantic features, overlooking the potential influence of positional information. Method: We conduct attribution analysis, attention and routing visualization, systematic ablation studies, and statistical analysis of expert activation distributions across token positions. Contribution/Results: We empirically demonstrate—across multiple state-of-the-art MoE architectures (e.g., Switch Transformer, GLaM)—that tokens at distinct positions exhibit strong, consistent preferences for specific experts, revealing a stable spatial bias in routing decisions. This phenomenon, termed “position-aware routing,” constitutes the first evidence that positional encoding critically shapes expert assignment in MoE models. Our findings establish a novel, interpretable phenomenological model of structured expert allocation and introduce position–semantics co-modeling as a principled optimization axis for MoE design—enhancing both routing efficiency and generalization capability.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

A common assumption is that MoE routers primarily leverage semantic features for expert selection. However, our study challenges this notion by demonstrating that positional token information also plays a crucial role in routing decisions. Through extensive empirical analysis, we provide evidence supporting this hypothesis, develop a phenomenological explanation of the observed behavior, and discuss practical implications for MoE-based architectures.

Problem

• Challenges assumption of semantic-only MoE routing
• Shows positional token info affects expert selection
• Provides empirical analysis and practical implications

Innovation

• MoE routers use positional token information
• Empirical analysis supports routing behavior hypothesis
• Practical implications for MoE architectures discussed