mixture of experts

Sparse Mixture-of-Experts (MoE) models often suffer from capacity waste and performance degradation due to routing bias toward a small subset of experts. Conventional load-balancing methods enforce uniform expert utilization but risk undermining semantic coherence, leading to knowledge redundancy across experts. To address this, we propose a similarity-preserving load-balancing mechanism: a differentiable routing loss grounded in token embedding similarity, which encourages semantically similar tokens to be consistently routed to the same expert—thereby jointly optimizing load distribution and routing consistency. Our approach requires no additional experts or auxiliary modules and integrates seamlessly into standard MoE training pipelines. Experiments demonstrate a 36% acceleration in convergence on benchmark tasks, substantial reduction in inter-expert knowledge redundancy, and improved model generalization and inference efficiency.

This work investigates the information-theoretic efficiency of routing mechanisms in sparse Mixture-of-Experts (MoE) architectures, aiming to balance model accuracy with communication and computational resource utilization. The gating router is modeled as a stochastic channel, and a discrete mutual information estimator is proposed under a finite expert pool. Empirical posterior distributions \( q(W|S) \) are leveraged to compute \( I(X;T) \) and \( I(S;W) \), with the latter shown to exhibit a monotonic relationship with the generalization gap. The Blahut–Arimoto algorithm is employed to trace the accuracy–rate trade-off curve. Experiments demonstrate that the proposed mutual information estimator effectively tracks the generalization gap and significantly outperforms both the Xu–Raginsky bound and the uniform joint bound, offering a practical analytical tool for resource-aware MoE systems.

To address router-induced load imbalance and accuracy degradation in Mixture-of-Experts (MoE) models, this work designs and systematically evaluates six router variants—including the novel MLP-Hadamard—enabling, for the first time, custom router replacement and end-to-end fine-tuning on quantized Qwen1.5-MoE. MLP-Hadamard introduces a structured sparse routing mechanism that enhances expert utilization while preserving high sparsity. Empirical analysis across BERT and Qwen1.5-MoE reveals that Linear routers incur the lowest latency, MLP and Attention routers offer superior expressivity, and MLP-Hadamard achieves the optimal trade-off among inference efficiency, load balancing, and parameter efficiency. This study establishes a reproducible benchmarking framework for MoE router design and delivers actionable insights for deploying efficient, production-ready MoE systems.

MoE models suffer from two key bottlenecks: imbalanced expert activation—causing parallel idle time and low utilization—and high communication overhead in expert parallelism. This paper proposes Collaborative-Constrained Routing (C2R), the first framework to establish an “expert collaboration–specialization” analytical paradigm. C2R explicitly regulates expert activation patterns via a collaboration-aware routing loss, a quantifiable expert specialization metric, lightweight gating regularization, and All2All communication optimization compatible with MegaBlocks—all without sacrificing model accuracy. Evaluated on LLaMA-MoE and Qwen-MoE, C2R improves downstream task performance by 0.51% and 0.33%, respectively, while significantly reducing inter-GPU All2All communication costs. Total training time decreases by 20–30% over state-of-the-art methods, achieving joint optimization of load balancing and communication efficiency.

This work addresses the limitations of conventional sparse mixture-of-experts (MoE) models, which employ independent routing at each layer, resulting in an excessively large path space and poor statistical efficiency that hinder the learning of stable expert routing structures. To overcome this, the authors propose Path-Constrained Mixture of Experts (PathMoE), a novel architecture that shares router parameters across layers to dramatically reduce the effective path space, thereby enhancing path consistency and structural learnability. Notably, PathMoE naturally induces token clustering according to linguistic functionality without requiring auxiliary load-balancing losses. Experiments demonstrate that PathMoE achieves lower perplexity, superior downstream task performance, and greater robustness to routing perturbations compared to standard MoE baselines, consistently across both 0.9B and 16B parameter scales.

This work addresses the lack of effective methods to evaluate whether experts in sparse Mixture-of-Experts (MoE) models achieve non-redundant specialization, particularly in small-scale settings. To this end, the authors construct the first benchmark that accurately reflects large-scale routing behavior at a smaller scale, integrating multi-domain distinguishable data and an ideal reference router grounded in domain definitions. They systematically compare multiple routing strategies and introduce quantitative metrics to assess expert specialization. Their experiments reveal that a “balanced routing regime” is crucial for achieving both high expert utilization and meaningful specialization, a finding they further validate on models up to 35 times larger, demonstrating strong scalability of their conclusions.

This work addresses the fundamental trade-off in sparse Mixture-of-Experts (MoE) models between load balancing and expert specialization, which often leads to routing collapse or diminished expert diversity. The authors propose Hi-MoE, a novel framework that decomposes routing into two coupled hierarchical levels: inter-group routing ensures balanced token distribution across expert groups, while intra-group routing fosters complementary expert specialization and prevents collapse. This principled redesign of router behavior consistently outperforms existing sparse routing and grouped MoE approaches across both NLP and vision benchmarks. In a 58B-token pretraining setting, Hi-MoE-7B achieves a 5.6% lower perplexity and 40% improved expert balance compared to OLMoE-7B.

This work addresses the storage and memory bottlenecks imposed by the large parameter counts of Mixture-of-Experts (MoE) models in edge-device deployment by proposing DECO, a sparse MoE architecture. DECO integrates differentiable ReLU-based routing, learnable expert-level scaling factors, NormSiLU activation functions, and non-gated MLP experts to achieve performance on par with dense Transformers while activating only 20% of the experts. Compared to existing MoE baselines, DECO substantially enhances sparsity stability and computational efficiency, delivering a 3× inference speedup over dense models on real hardware. The design thus achieves an effective balance among high performance, low computational overhead, and minimal memory footprint.

This work proposes a novel mixture-of-experts (MoE) architecture that eliminates the need for explicit routing mechanisms commonly found in traditional MoE models. By embedding activation logic directly within each expert and enabling end-to-end continuous gradient flow, experts autonomously determine their own activation without reliance on external routers, Softmax operations, Top-K selection, or hard-coded load-balancing heuristics. The approach introduces a unified, adaptive load-balancing framework that jointly optimizes resource allocation across both experts and tokens, supporting configurable dual-objective balancing. Experimental results demonstrate that the proposed model consistently outperforms existing baselines across multiple benchmarks, exhibiting superior scalability and robustness while removing rigid inductive biases imposed by centralized routing.

This work addresses the routing collapse and expert deadlocks that commonly afflict Token-Choice sparse Mixture-of-Experts (MoE) architectures in video diffusion Transformers, which severely limit expert diversity utilization. Starting from a 5-billion-parameter dense model, the authors formulate three principles for converting dense networks to MoE. Through temporal routing analysis of 65 million tokens, they reveal that deadlocked layers follow a U-shaped distribution across the network depth and propose a “functional redundancy” hypothesis to explain this phenomenon. Building on these insights, they integrate expert cloning, zero-initialized gating, auxiliary losses, and enhanced router designs—including linear, MLP, and cross-attention variants—to effectively mitigate bfloat16 precision pitfalls. Their approach alleviates single-expert deadlocks in approximately two-thirds of network layers, endows the model with partial self-recovery capability, delineates the capacity limits of the Token-Choice paradigm, and outlines a three-stage roadmap toward unified vision models and ultimately world models.