Boundary Mass and the Soft-to-Hard Limit in Mixture-of-Experts

2026-05-03

AI Summary

This work investigates the singular behavior of Softmax routing in mixture-of-experts models as the temperature approaches zero, where soft routing transitions to hard routing. Focusing on the “boundary mass”—the probability mass near inputs for which the top two routing scores are nearly equal—it demonstrates that the zero-temperature limit is governed by a thin geometric layer around the routing decision boundary in input space. By introducing the notion of boundary mass and establishing its quantitative link to the geometry of routing interfaces—via the coarea formula, tubular neighborhood estimates, Γ-convergence theory, and transversality conditions—the authors prove that the soft objective Γ-converges to its hard-routing counterpart. Within a teacher-student framework, they further elucidate landscape transition and local symmetry-breaking mechanisms, yielding quantitative bounds on soft–hard risk discrepancy, alignment of symmetry breaking with the teacher’s separating hyperplane in the two-expert Gaussian case, and the ability of low-temperature soft routing to approximately recover teacher parameters while inheriting strict saddle structures.

Abstract

Softmax-routed mixture-of-experts models approach hard routing as the temperature tends to zero, but this limit is singular near routing ties. This paper studies that singularity at the population level for squared-loss MoE regression. The central object is the \emph{boundary mass}, namely the probability that the top two router scores are separated by only a small margin. Under smoothness and transversality assumptions on the router and input law, we prove coarea/tube estimates showing that this mass is linear in the slab width, with leading constant given by a surface integral over the routing interface in the binary case. These estimates yield quantitative soft-to-hard risk bounds and, under compactness and uniform margin control, $Γ$-convergence of the soft objectives to the hard-routing objective. The main conclusion is that the zero-temperature limit is controlled by a thin geometric layer around routing interfaces, not by the full input space. We then use this geometric core in two more model-dependent directions. In a teacher--student setting, we prove a conditional landscape-transfer principle showing that, when the profiled hard-routing problem has favorable identifiability and curvature and the relevant derivatives transfer at boundary-layer scale, small-temperature soft routing inherits approximate teacher recovery and strict-saddle behavior away from teacher-equivalent partitions. We also give a reduced two-expert Gaussian calculation that illustrates a local symmetry-breaking mechanism aligned with the teacher separator.

Problem

• Mixture-of-Experts
• soft-to-hard limit
• boundary mass
• routing singularity
• zero-temperature limit

Innovation

• boundary mass
• soft-to-hard routing
• Γ-convergence
• mixture-of-experts
• geometric layer