Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts

2026-09-01

AI Summary

为解决Mixture-of-Experts架构中专家专业化受限问题，提出对比路由机制（CoRM），通过对比每个token与隐藏状态的移动平均值来改进路由信号，提高零样本推理性能。

Abstract

In current Mixture-of-Experts architectures, routing is performed based on representations dominated by structure shared across all tokens, limiting expert specialization. We show that contrasting each token against an Exponential Moving Average of the layer's hidden states, rather than routing on absolute magnitude, concentrates the routing signal onto a low-dimensional, highly separable subspace. Building on this, we propose the Contrastive Routing Mechanism (CoRM), which scores each expert by the gap between its affinity for the incoming token and its affinity for this shared reference state, interpreted through a distinct per-expert projection. The resulting experts have routing boundaries that align with linguistic structure significantly more than the Top-k baseline. Our experiments show that CoRM improves average zero-shot accuracy by +0.67 to +1.69 points (Top-1) and +1.38 to +1.77 points (Top-2) over standard Top-k MoE baselines on nine zero-shot reasoning benchmarks, at the minimal cost of 2.9% added parameters and 2.6% added FLOPs per token.

Problem

• Mixture-of-Experts
• routing
• expert specialization
• linguistic structure
• zero-shot reasoning

Innovation

• Contrastive Routing
• Mixture-of-Experts
• Exponential Moving Average
• Zero-shot Accuracy
• Linguistic Structure