Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control

2026-06-21

AI Summary

This work investigates whether identical tokens in sparse Mixture-of-Experts (MoE) language models consistently correspond to the same routing states and reasoning behaviors, and proposes a general-purpose reasoning control method that does not rely on answer strings. By analyzing routing states at fixed token positions, the authors find that these states effectively distinguish task contexts, reasoning trajectories, and patterns of reasoning effort. Building on this insight, they introduce Routing Agreement Decoding (RAD), a framework that leverages weighted Jaccard similarity, K-nearest neighbor clustering, and anchor window analysis to select outputs based on routing consistency. Experiments across ten MoE configurations and six datasets demonstrate that RAD matches the performance of majority voting on mathematical reasoning, GPQA, and code tasks, while also enabling effective decoding in settings without ground-truth answer strings—such as code generation and patch selection—thereby revealing, for the first time, the rich semantic structure embedded in MoE routing states.

Abstract

In sparse Mixture-of-Experts language models, does the same token id imply the same router state and the same experts producing it? Holding the emitted token id fixed at repeated anchors, we find it does not: the experts that produce it still separate task context, trajectory history, and reasoning-effort mode. This residual structure supports test-time control: near \emph{boundary} anchors (the final-response transition) and \emph{delimiter} anchors (which open the answer, e.g.\ \texttt{\textbackslash boxed\{} or code fences), routing neighborhoods already align with final-answer basins at a marker-only readout and strongest when the routing is read at the answer opening. We operationalize this as \textbf{RAD} (Routing Agreement Decoding), an answer-string-free multi-rollout selector: it locates a fixed anchor, represents each rollout by its anchor-window MoE routing states, and returns the densest Weighted-Jaccard $K$-NN route-basin center, without parsing, normalizing, executing, or voting over answer strings. Across 10 sparse-MoE configurations (gpt-oss, Qwen3-MoE) and 6 datasets spanning math, GPQA, and code, RAD is on par with Majority where string voting is well-posed, with small positive paired deltas (RAD $73.9$ / RAD+DC $74.2$ vs.\ Majority $73.6$). Like majority voting, RAD is not a verifier: a dense \emph{wrong} basin can still win. Its value is the interface: the same selector gives direct pass@1 on code, where exact-string voting is ill-defined, and the same routing-density principle, re-anchored to the agentic boundary, improves best-of-16 patch selection on SWE-bench Verified over random, where patches have no answer string to vote on.

Problem

• Mixture-of-Experts
• token routing
• reasoning control
• sparse models
• expert selection

Innovation

• Mixture-of-Experts
• routing control
• reasoning alignment
• answer-free decoding
• RAD