Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts

2026-06-17

AI Summary

This work addresses the discontinuity in Sparse Mixture-of-Experts (SMoE) models arising from the Top-k expert selection mechanism, which can cause abrupt output changes as nearby inputs activate different expert subsets. The paper presents the first systematic geometric and stochastic analysis of this discontinuity, employing measure-theoretic slicing, diffusion process modeling, and first-passage time analysis to demonstrate that lower-order discontinuity manifolds dominate in high-dimensional spaces and that random trajectories almost surely encounter first-order discontinuities first. Building on these insights, the authors propose a lightweight local smoothing mechanism that substantially enhances model continuity and performance across both language and vision tasks while maintaining computationally tractable overhead.

Abstract

Sparse Mixture-of-Experts (SMoE) architectures are now widely deployed in state-of-the-art language and vision models, where conditional routing allows scaling to very large networks. However, this very Top-$k$ expert selection that enables conditional routing also renders the SMoE map inherently discontinuous. In the vicinity of these discontinuity surfaces, even inputs that are arbitrarily close may activate substantially different sets of experts resulting in significantly different outputs. In this work we give a rigorous geometric and stochastic analysis of these discontinuities. We first classify them by order, determined by the number of tied experts at a switching event. Using measure-theoretic slicing arguments, we establish asymptotic volume estimates for the thickened discontinuity surfaces, showing that lower-order discontinuity sets dominate, whereas higher-order ones occupy a vanishingly small relative volume. Next, modeling random perturbations in the input space via a diffusion process, we prove that the path eventually encounter a discontinuity, and moreover that the first hit almost surely occurs on an order-1 discontinuity with explicit finite-time probability bounds. We further derive occupation-time bounds that quantify the duration the random path spend in the neighborhoods of each discontinuity order. These theoretical results imply that inputs are more likely to lie near lower order discontinuities. Motivated by this insight, we propose a simple smoothing mechanism that can be directly applied to existing SMoEs, softly incorporating experts near discontinuities; our analysis guarantees that the added computational overhead remains small while providing localized smoothing near discontinuities, and experiments across language and vision tasks show that smoothing not only enforces continuity of the SMoE map but also enhances empirical performance.

Problem

• Sparse Mixture-of-Experts
• discontinuities
• conditional routing
• Top-k selection
• expert activation

Innovation

• Sparse Mixture-of-Experts
• discontinuity analysis
• geometric measure theory
• stochastic diffusion
• expert smoothing