Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function

2026-02-01

AI Summary

This work addresses unresolved challenges in sigmoid-gated Mixture-of-Experts (MoE) models for classification tasks—namely, poor convergence, low sample efficiency, and undesirable coupling between the temperature parameter and gating dynamics. The authors propose an improved sigmoid gating mechanism that, for the first time, provably outperforms softmax gating in multi-class settings. By replacing the inner product with a Euclidean distance-based scoring function, the method effectively decouples the temperature from gating parameters, leading to markedly improved optimization dynamics. Theoretical analysis demonstrates that the approach reduces sample complexity from exponential to polynomial in both expert selection and parameter estimation, substantially lowering the data requirements and thereby enhancing model scalability and training efficiency.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks to language modeling. Additionally, recent efforts have demonstrated that the sigmoid gate is provably more sample-efficient than its softmax counterpart under regression settings. Nevertheless, there are three notable concerns that have not been addressed in the literature, namely (i) the benefits of the sigmoid gate have not been established under classification settings; (ii) existing sigmoid-gated MoE models may not converge to their ground-truth; and (iii) the effects of a temperature parameter in the sigmoid gate remain theoretically underexplored. To tackle these open problems, we perform a comprehensive analysis of multinomial logistic MoE equipped with a modified sigmoid gate to ensure model convergence. Our results indicate that the sigmoid gate exhibits a lower sample complexity than the softmax gate for both parameter and expert estimation. Furthermore, we find that incorporating a temperature into the sigmoid gate leads to a sample complexity of exponential order due to an intrinsic interaction between the temperature and gating parameters. To overcome this issue, we propose replacing the vanilla inner product score in the gating function with a Euclidean score that effectively removes that interaction, thereby substantially improving the sample complexity to a polynomial order.

Problem

• mixture-of-experts
• sigmoid gating
• classification
• model convergence
• temperature parameter

Innovation

• Mixture of Experts
• Sigmoid Gating
• Sample Complexity
• Temperature Parameter
• Euclidean Score