SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment

2026-06-24

AI Summary

This work addresses the challenge that low-resource languages struggle to effectively share expert knowledge in sparse Mixture-of-Experts (MoE) models due to significant discrepancies in routing distributions compared to high-resource languages, which limits multilingual performance. To mitigate this, the authors propose a semantics-anchored routing alignment mechanism that achieves cross-lingual alignment within the MoE’s routing layer for the first time. By treating high-resource languages as semantic anchors and constraining the symmetric Jensen–Shannon divergence, the method guides low-resource language inputs toward consistent expert pathways. This approach substantially enhances cross-lingual expert sharing consistency, yielding Global-MMLU improvements of 0.8% and 1.2% on Qwen3-30B-A3B and Phi-3.5-MoE-instruct, respectively, and effectively boosts performance on low-resource languages.

Abstract

Sparse Mixture-of-Experts (MoE) architectures have emerged as an increasingly influential paradigm as they offer a strategic balance between parameter scalability and computational efficiency. However, low-resource languages, which suffer from a scarcity of high-quality training data, often have their tokens routed to different experts than those predominantly activated by high-resource inputs, which limits cross-lingual expert sharing. This cross-lingual routing divergence consequently hinders their efficacy in multilingual contexts. To address this issue, we propose SARA (Semantically Anchored Routing Alignment), a framework designed to transfer specialized capabilities from high-resource languages as anchors to low-resource languages. SARA explicitly aligns the routing distribution of multilingual inputs with high-resource semantic anchors using a symmetric Jensen-Shannon (JS) divergence constraint. Unlike traditional distillation methods that operate on output logits, SARA directly aligns the internal routing distributions of MoE layers, encouraging mechanistic consistency in expert selection across languages. We conduct experiments on 2 LLMs across 5 low-resource languages and 3 benchmarks. Experiment results demonstrate that SARA outperforms standard instruction tuning, e.g., +0.8% on Qwen3-30B-A3B and +1.2% on Phi-3.5-MoE-instruct on Global-MMLU. Further analyses show that SARA effectively addresses performance bottlenecks in low-resource languages, providing a scalable pathway to enhance multilingual capabilities in sparse architectures.

Problem

• Mixture-of-Experts
• multilingual
• low-resource languages
• routing divergence
• cross-lingual sharing

Innovation

• Mixture-of-Experts
• multilingual alignment
• routing distribution
• semantic anchoring
• low-resource languages