Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations

2026-04-15

AI Summary

Sparse Mixture-of-Experts (MoE) models often suffer from hallucinations on long-tail tasks due to their static Top-k routing mechanism, which overlooks experts encoding critical factual knowledge. This work proposes Counterfactual Routing (CoR), a training-free inference framework that dynamically activates causally relevant yet initially omitted experts by leveraging inter-layer perturbation analysis and a novel Counterfactual Expert Influence (CEI) metric. CoR introduces counterfactual reasoning into MoE routing for the first time, precisely identifying and engaging knowledge-intensive experts without increasing computational overhead or altering the total number of activated experts per token. Experimental results demonstrate that CoR improves factual accuracy by an average of 3.1% across TruthfulQA, FACTOR, and TriviaQA, significantly outperforming existing static expansion strategies.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Reasoning under Uncertainty: Causality
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Sparse Mixture-of-Experts (MoE) models have achieved remarkable scalability, yet they remain vulnerable to hallucinations, particularly when processing long-tail knowledge. We identify that this fragility stems from static Top-$k$ routing: routers tend to favor high-frequency patterns over rare factual associations. Consequently, ``specialist experts'' possessing critical long-tail knowledge are often assigned low gating scores and remain ``dormant'' -- under-prioritized for specific tokens despite their proven causal importance on other inputs. To address this, we propose Counterfactual Routing (CoR), a training-free inference framework designed to awaken these dormant experts. CoR integrates layer-wise perturbation analysis with the Counterfactual Expert Impact (CEI) metric to dynamically shift computational resources from syntax-dominant to knowledge-intensive layers while maintaining a constant total activation count, effectively retrieving causally decisive experts via virtual ablation. Extensive experiments on TruthfulQA, FACTOR, and TriviaQA demonstrate that CoR improves factual accuracy by 3.1\% on average without increasing the inference budget, establishing a superior Pareto frontier compared to static scaling strategies.

Problem

• Mixture-of-Experts
• hallucinations
• long-tail knowledge
• expert routing
• factual accuracy

Innovation

• Counterfactual Routing
• Mixture-of-Experts
• Hallucination Mitigation
• Dormant Experts
• Knowledge-intensive Inference