mixture of experts

Existing PEFT methods neglect the dynamic expert routing mechanism inherent in Mixture-of-Experts (MoE) models, leading to architectural misalignment between adaptation modules and the underlying MoE structure. To address this, we propose *Routed-PEFT*, the first PEFT framework that explicitly incorporates expert routing into adapter design—dynamically assigning dedicated adapters to individual experts, thereby enabling joint optimization of expert specialization and task-specific adaptation. We systematically evaluate Routed-PEFT on OLMoE and Mixtral architectures, integrating it with LoRA, Adapter, and other PEFT variants under diverse routing strategies across commonsense and mathematical reasoning benchmarks. With only 0.1%–0.5% additional trainable parameters, Routed-PEFT achieves average accuracy gains of 2.3–5.7 percentage points over standard PEFT baselines. Moreover, our analysis uncovers task-dependent optimal routing configurations, establishing a novel paradigm for efficient fine-tuning of MoE models.

This work systematically dissects the multidimensional design space of Mixture-of-Experts (MoE) architectures in large language models, moving beyond conventional generational narratives. It introduces a five-dimensional analytical framework encompassing expert granularity, topology, routing flexibility, load-balancing scope, and execution structure, and constructs a dependency graph to elucidate the coupling mechanisms across four control planes: expert topology, routing, load balancing, and expert parallelism. The framework’s validity is empirically demonstrated through iso-budget pretraining experiments integrating algorithmic innovations—such as Top-k routing, shared and fine-grained experts, and dynamic expert composition—with system-level optimizations including token dispatch, device placement, and all-to-all communication. The study further distills key open challenges for the future development of MoE systems.

This work addresses a key limitation in sparse Mixture-of-Experts (MoE) models, where the routing mechanism jointly handles expert selection and output weighting, potentially constraining performance. The study provides the first systematic validation that these two functions should be decoupled and introduces Fixed Dispatch with Adaptive Aggregation (FDAA): a lightweight, learnable aggregation head is added atop a frozen backbone and fixed expert assignments, enabling end-to-end optimization of aggregation weights via the language modeling objective. Evaluated on pretrained MoE models such as OLMoE and DeepSeek-V2-Lite, FDAA achieves a 0.1523 reduction in cross-entropy on WikiText-103 and demonstrates consistent improvements across diverse benchmarks including C4 and PTB, confirming both the efficacy and generality of the proposed decoupling strategy.

MoE models suffer from two key bottlenecks: imbalanced expert activation—causing parallel idle time and low utilization—and high communication overhead in expert parallelism. This paper proposes Collaborative-Constrained Routing (C2R), the first framework to establish an “expert collaboration–specialization” analytical paradigm. C2R explicitly regulates expert activation patterns via a collaboration-aware routing loss, a quantifiable expert specialization metric, lightweight gating regularization, and All2All communication optimization compatible with MegaBlocks—all without sacrificing model accuracy. Evaluated on LLaMA-MoE and Qwen-MoE, C2R improves downstream task performance by 0.51% and 0.33%, respectively, while significantly reducing inter-GPU All2All communication costs. Total training time decreases by 20–30% over state-of-the-art methods, achieving joint optimization of load balancing and communication efficiency.

This study investigates the true origins of expert specialization in Mixture-of-Experts (MoE) models, challenging the assumption that routing mechanisms reflect genuine domain-specific expertise. Through theoretical analysis and empirical evaluation, the work demonstrates for the first time that expert usage similarity is entirely determined by the geometric structure of hidden states rather than architecture-induced specialization. It further reveals that load-balancing losses suppress directions corresponding to shared representations, leading to a phenomenon termed “specialization collapse.” Additionally, routing patterns in pretrained MoEs prove largely semantically uninterpretable. Combining linear mapping analysis, geometric modeling of hidden states, cross-model and cross-layer routing comparisons, and theoretical derivation of loss functions, the study validates across five pretrained models a strong alignment between routing behavior and representation-space geometry: expert activations exhibit high overlap even on semantically unrelated inputs, and prompt-level routing fails to predict expert selection during actual inference.

Existing approaches struggle to disentangle load balancing from functional specialization in Mixture-of-Experts (MoE) models, hindering a deeper understanding of expert mechanisms. This work proposes DBES, a diagnostic framework that, for the first time, moves beyond accuracy-based evaluation and systematically quantifies the degree of expert functional specialization through five theoretically grounded metrics: Routing Specialization, Normalized Effective Rank, Domain Isolation, Routing Stiffness Score, and N-gram Expertise. Leveraging this diagnostic insight, we design an intervention-based post-training strategy that, using only 15% of the original training resources, selectively activates highly specialized expert pathways, achieving performance gains of 66% to 94.48% on domain-specific tasks. Our analysis further reveals distinct specialization paradigms across models and offers actionable pathways for targeted optimization.

This study investigates the relationship between safety behaviors and expert routing mechanisms in aligned Mixture-of-Experts (MoE) large language models. The authors find that a model’s safety capabilities can be concentrated in a small subset of experts and are largely independent of the routing policy, rather than being driven by dedicated refusal-oriented experts. To leverage this insight, they propose the Router-Agnostic Safety-critical Expert Tuning (RASET) framework, which integrates a contrastive routing sensitivity criterion with parameter-efficient fine-tuning to precisely identify and optimize safety-critical experts without altering the original routing behavior. Experiments demonstrate that RASET significantly steers model safety outputs with minimal semantic interference, revealing for the first time the existence of localized, manipulable expert-level safety mechanisms—and their potential vulnerabilities—within MoE architectures.

This work addresses the disconnect in existing Mixture-of-Experts (MoE) models between shared computation and dynamic routing, which overlooks the interdependence between reusable computation and residual expert requirements. The paper proposes UniF-MoE, a unified framework introducing a novel “shared-first, routed-later” mechanism: it first processes common features through a shared general-purpose module and then dynamically activates residual experts based on a shared-demand score and complementarity. Key innovations include key prototype selection, cumulative routing quality allocation, and Gram regularization to enhance routing sparsity and diversity, revealing a negative correlation between shared coverage and residual demand. Experiments demonstrate that UniF-MoE outperforms both static and dynamic MoE approaches on DomainBed and GLUE benchmarks while significantly reducing activated computation, inference latency, and memory footprint.

This work addresses the lack of effective design principles for routers in existing Mixture-of-Experts (MoE) models, which struggle to accurately capture the affinity between tokens and experts. The authors propose, for the first time, using the dominant singular directions of expert matrices as the target for router design and introduce a novel “power iteration followed by shrinkage” paradigm. During pretraining, they employ manifold optimization to dynamically align the router’s row vectors with these dominant singular directions. This approach achieves a favorable balance among alignment accuracy, computational efficiency, and training stability. Experiments on MoE models ranging from 1B to 11B parameters demonstrate substantial performance improvements, validating the effectiveness of the proposed router redesign strategy.

This work addresses the pervasive issue of deep routing collapse in large Mixture-of-Experts (MoE) models for low-resource languages, which leads to imbalanced expert utilization and constrained multilingual capabilities. The study reveals, for the first time, that this phenomenon stems from insufficient pretraining data rather than inherent linguistic properties. To diagnose multilingual capacity, the authors propose routing entropy and expert specialization as key indicators. Through balanced bilingual continual pretraining (CPT) and supervised fine-tuning (SFT) on Hebrew, Japanese, and other languages using both pure Transformer and Mamba-Transformer hybrid architectures, they demonstrate that CPT substantially increases routing entropy, encourages language-agnostic expert sharing, and consistently enhances downstream performance, whereas SFT yields limited gains. These findings underscore the critical role of data balance in scaling MoE models multilingually.