conditional routing

This work investigates the information-theoretic efficiency of routing mechanisms in sparse Mixture-of-Experts (MoE) architectures, aiming to balance model accuracy with communication and computational resource utilization. The gating router is modeled as a stochastic channel, and a discrete mutual information estimator is proposed under a finite expert pool. Empirical posterior distributions \( q(W|S) \) are leveraged to compute \( I(X;T) \) and \( I(S;W) \), with the latter shown to exhibit a monotonic relationship with the generalization gap. The Blahut–Arimoto algorithm is employed to trace the accuracy–rate trade-off curve. Experiments demonstrate that the proposed mutual information estimator effectively tracks the generalization gap and significantly outperforms both the Xu–Raginsky bound and the uniform joint bound, offering a practical analytical tool for resource-aware MoE systems.

This work proposes a novel mixture-of-experts (MoE) architecture that eliminates the need for explicit routing mechanisms commonly found in traditional MoE models. By embedding activation logic directly within each expert and enabling end-to-end continuous gradient flow, experts autonomously determine their own activation without reliance on external routers, Softmax operations, Top-K selection, or hard-coded load-balancing heuristics. The approach introduces a unified, adaptive load-balancing framework that jointly optimizes resource allocation across both experts and tokens, supporting configurable dual-objective balancing. Experimental results demonstrate that the proposed model consistently outperforms existing baselines across multiple benchmarks, exhibiting superior scalability and robustness while removing rigid inductive biases imposed by centralized routing.

This work investigates the trade-off between model expressivity and generalization performance in Mixture-of-Experts (MoE) architectures under communication constraints. For the first time, rate-distortion theory is introduced into MoE analysis by modeling the gating mechanism as a stochastic channel operating at a finite rate. By integrating mutual information-based generalization bounds with the rate-distortion function \(D(R_g)\), the study establishes a quantitative relationship between the gating communication rate and generalization error. A theoretical upper bound on generalization error is derived and validated through synthetic multi-expert model simulations, which demonstrate that reducing the gating rate, while limiting expressivity, can enhance generalization. Based on these insights, the paper proposes a capacity-aware design principle for MoE systems, offering theoretical guidance for efficient model construction in resource-constrained settings.

In conventional sparse Mixture-of-Experts (MoE) models, the Top-K routing mechanism is non-differentiable, impeding end-to-end optimization and limiting scalability. This paper proposes the first fully differentiable MoE architecture, replacing the discrete Top-K + Softmax routing with a continuous, ReLU-based differentiable routing mechanism. The method incorporates sparsity regularization and load-balancing constraints to ensure controlled expert activation and equitable expert utilization, while enabling dynamic expert selection and domain specialization. Extensive experiments across varying model scales, numbers of experts, and granularity levels demonstrate consistent and significant improvements over baseline Top-K MoE—particularly in expert scalability and downstream task performance. The architecture maintains high sparsity without sacrificing trainability or inference efficiency. Code is publicly available.

This work addresses two key limitations in mixture-of-experts (MoE) models: the lack of theoretical connection between MoE routing and self-attention, and the low sample efficiency of linear gating. We propose quadratic gating—replacing conventional linear routing with a quadratic function—and establish, for the first time, its rigorous equivalence to self-attention. Leveraging this equivalence, we derive principled design criteria for optimal quadratic gating and expert functions, leading to a novel high-performance attention mechanism. Theoretically, via statistical learning analysis, we prove that quadratic gating substantially enhances the expressivity and parameter/sample efficiency of expert selection. Empirically, our MoE variant outperforms linear-gating baselines across multiple tasks; the new attention mechanism surpasses state-of-the-art methods—including FlashAttention and Multi-Head Attention—while exhibiting strong alignment between theoretical predictions and empirical results. The framework thus achieves both interpretability and practical efficacy.

This work addresses the “trust region collapse” problem in multi-model routing, where tight coupling between inference and routing leads to systematic suppression of high-capability experts. To resolve this, the authors propose EntroRouter, a single-round routing framework that decouples inference from routing by adopting entropy regularization as its core objective. The method introduces a high-entropy prior for exploration and a soft-anchor mechanism based on offline capability estimation, combined with soft-supervision initialization and a controlled entropy contraction strategy. These innovations effectively prevent strong experts from being underutilized. Experimental results demonstrate that EntroRouter reduces computational overhead by 48.25% while preserving 98.3% of the top expert’s accuracy.

This work addresses the inefficiency of traditional Mixture-of-Experts (MoE) architectures, which rely on static Top-k routing and cannot dynamically adjust the number of activated experts per input. Framing token routing as an information encoding task, the study establishes a novel connection between gating entropy and the Minimum Description Length (MDL) principle, leading to an uncertainty-aware adaptive routing mechanism that explicitly balances model complexity against performance. Evaluated across multiple backbone networks and benchmarks, the proposed method achieves a 36.5% improvement in expert activation sparsity while maintaining 99.5% of the original static routing performance, significantly outperforming existing static and heuristic dynamic routing strategies.

This work addresses a key limitation in sparse Mixture-of-Experts (MoE) models, where the routing mechanism jointly handles expert selection and output weighting, potentially constraining performance. The study provides the first systematic validation that these two functions should be decoupled and introduces Fixed Dispatch with Adaptive Aggregation (FDAA): a lightweight, learnable aggregation head is added atop a frozen backbone and fixed expert assignments, enabling end-to-end optimization of aggregation weights via the language modeling objective. Evaluated on pretrained MoE models such as OLMoE and DeepSeek-V2-Lite, FDAA achieves a 0.1523 reduction in cross-entropy on WikiText-103 and demonstrates consistent improvements across diverse benchmarks including C4 and PTB, confirming both the efficacy and generality of the proposed decoupling strategy.

This work addresses the inefficiency of conventional Mixture-of-Experts (MoE) models, which employ a fixed top-k expert selection strategy that fails to dynamically allocate computational resources according to individual token demands. To overcome this limitation, the authors propose a training-free, plug-in method for inference that introduces, for the first time in MoE architectures, an elbow-point detection mechanism. By analyzing the probability distribution output by the router, this approach adaptively determines the number of experts to activate per token. Integrating principles from ranking and load balancing theory, the method achieves dynamic resource allocation while preserving balanced expert utilization. Experimental results demonstrate that the proposed technique reduces average inference latency by 5.3% across mainstream MoE models without compromising accuracy on six benchmark evaluations.

This work addresses the disconnect in existing Mixture-of-Experts (MoE) models between shared computation and dynamic routing, which overlooks the interdependence between reusable computation and residual expert requirements. The paper proposes UniF-MoE, a unified framework introducing a novel “shared-first, routed-later” mechanism: it first processes common features through a shared general-purpose module and then dynamically activates residual experts based on a shared-demand score and complementarity. Key innovations include key prototype selection, cumulative routing quality allocation, and Gram regularization to enhance routing sparsity and diversity, revealing a negative correlation between shared coverage and residual demand. Experiments demonstrate that UniF-MoE outperforms both static and dynamic MoE approaches on DomainBed and GLUE benchmarks while significantly reducing activated computation, inference latency, and memory footprint.