token routing

This work systematically investigates the interplay among key design dimensions in Mixture-of-Experts (MoE) architectures—such as the number of experts, expert granularity, heterogeneity, shared experts, and load balancing—through over 2,000 large-scale pretraining experiments. The study reveals that the number of experts and their granularity are the dominant factors governing model performance, while other design choices exert comparatively limited influence. Notably, increasing the total MoE parameters consistently enhances performance across all active parameter budgets, and the optimal expert size is determined solely by the number of active parameters. Furthermore, the effectiveness of dropless routing is empirically validated, demonstrating consistent performance gains.

This work systematically dissects the multidimensional design space of Mixture-of-Experts (MoE) architectures in large language models, moving beyond conventional generational narratives. It introduces a five-dimensional analytical framework encompassing expert granularity, topology, routing flexibility, load-balancing scope, and execution structure, and constructs a dependency graph to elucidate the coupling mechanisms across four control planes: expert topology, routing, load balancing, and expert parallelism. The framework’s validity is empirically demonstrated through iso-budget pretraining experiments integrating algorithmic innovations—such as Top-k routing, shared and fine-grained experts, and dynamic expert composition—with system-level optimizations including token dispatch, device placement, and all-to-all communication. The study further distills key open challenges for the future development of MoE systems.

This work challenges the prevailing assumption that Mixture-of-Experts (MoE) models achieve domain specialization through sparse routing, introducing the COMMITTEEAUDIT framework to systematically analyze expert-level routing behavior. Through quantitative and qualitative evaluation of multiple representative MoE models on the MMLU benchmark, we uncover the existence of persistent “standing committees”—a small subset of experts that consistently dominate routing weights across domains and layers, regardless of routing budget constraints. These core experts anchor structural and syntactic reasoning, while peripheral experts handle only narrow, domain-specific knowledge. Our findings reveal that the actual degree of specialization in MoE models is substantially lower than commonly assumed and suggest that current load-balancing training objectives may conflict with the model’s intrinsic optimization dynamics.

This work addresses the fundamental trade-off in sparse Mixture-of-Experts (MoE) models between load balancing and expert specialization, which often leads to routing collapse or diminished expert diversity. The authors propose Hi-MoE, a novel framework that decomposes routing into two coupled hierarchical levels: inter-group routing ensures balanced token distribution across expert groups, while intra-group routing fosters complementary expert specialization and prevents collapse. This principled redesign of router behavior consistently outperforms existing sparse routing and grouped MoE approaches across both NLP and vision benchmarks. In a 58B-token pretraining setting, Hi-MoE-7B achieves a 5.6% lower perplexity and 40% improved expert balance compared to OLMoE-7B.

This work investigates the opaque expert specialization mechanism in Mixture-of-Experts (MoE) models, which limits inference and memory efficiency. By analyzing domain-specific routing patterns and employing an early-decoding framework, the study systematically examines how individual experts contribute to model outputs. Through comprehensive analyses—including routing distribution statistics, cosine similarity of hidden states, comparisons between single-expert and ensemble outputs, and perplexity evaluation—the authors find that a small subset of experts handles over 50% of all requests. Remarkably, outputs from a single dominant expert exhibit high consistency with the full model (cosine similarity up to 0.95), with only a 5% increase in perplexity. These findings suggest that precise expert pruning can substantially enhance inference efficiency without compromising performance, offering a promising avenue for efficient MoE deployment and knowledge localization.

This work proposes a novel mixture-of-experts (MoE) architecture that eliminates the need for explicit routing mechanisms commonly found in traditional MoE models. By embedding activation logic directly within each expert and enabling end-to-end continuous gradient flow, experts autonomously determine their own activation without reliance on external routers, Softmax operations, Top-K selection, or hard-coded load-balancing heuristics. The approach introduces a unified, adaptive load-balancing framework that jointly optimizes resource allocation across both experts and tokens, supporting configurable dual-objective balancing. Experimental results demonstrate that the proposed model consistently outperforms existing baselines across multiple benchmarks, exhibiting superior scalability and robustness while removing rigid inductive biases imposed by centralized routing.

This work addresses the disconnect in existing Mixture-of-Experts (MoE) models between shared computation and dynamic routing, which overlooks the interdependence between reusable computation and residual expert requirements. The paper proposes UniF-MoE, a unified framework introducing a novel “shared-first, routed-later” mechanism: it first processes common features through a shared general-purpose module and then dynamically activates residual experts based on a shared-demand score and complementarity. Key innovations include key prototype selection, cumulative routing quality allocation, and Gram regularization to enhance routing sparsity and diversity, revealing a negative correlation between shared coverage and residual demand. Experiments demonstrate that UniF-MoE outperforms both static and dynamic MoE approaches on DomainBed and GLUE benchmarks while significantly reducing activated computation, inference latency, and memory footprint.

This work addresses the challenges of weak generalization, expert redundancy, and degraded performance on rare domains in multi-dataset joint training, which stem from distribution shifts and semantic inconsistencies in labels. To this end, the authors propose the GEM framework, which replaces the learnable router in conventional Mixture-of-Experts (MoE) architectures with a planner–compiler design. GEM introduces a deterministic expert assignment mechanism based on linear programming relaxation and hierarchical rounding, strictly enforcing capacity constraints without relying on load-balancing losses. This approach significantly enhances expert specialization and routing interpretability. Integrated with a DINO vision backbone, GEM-DINO achieves state-of-the-art performance on the UODB benchmark, markedly improving results on rare datasets and effectively mitigating task interference in few-shot adaptation scenarios.

This work addresses the training challenges in Mixture-of-Experts (MoE) models caused by the non-differentiability of top-k routing. To overcome this, the authors propose ProbMoE, a framework that formulates expert selection as a probability distribution over discrete subsets under a cardinality constraint, thereby enabling differentiable routing. During forward propagation, exactly k experts are activated via constrained sampling, while backward propagation employs marginal probability gradients over the subset space as unbiased surrogates for true gradients. This approach enables, for the first time, probabilistic end-to-end training with exact k-expert routing and naturally extends to dynamic-k routing, allowing per-token adaptive expert assignment. Experiments demonstrate that the Exact-k variant significantly improves expert utilization and routing diversity, whereas the Dynamic-k variant achieves comparable performance with fewer activated experts.

Existing approaches struggle to disentangle load balancing from functional specialization in Mixture-of-Experts (MoE) models, hindering a deeper understanding of expert mechanisms. This work proposes DBES, a diagnostic framework that, for the first time, moves beyond accuracy-based evaluation and systematically quantifies the degree of expert functional specialization through five theoretically grounded metrics: Routing Specialization, Normalized Effective Rank, Domain Isolation, Routing Stiffness Score, and N-gram Expertise. Leveraging this diagnostic insight, we design an intervention-based post-training strategy that, using only 15% of the original training resources, selectively activates highly specialized expert pathways, achieving performance gains of 66% to 94.48% on domain-specific tasks. Our analysis further reveals distinct specialization paradigms across models and offers actionable pathways for targeted optimization.