Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models

2026-05-27

AI Summary

This work addresses the performance imbalance in multilingual tasks when fine-tuning Mixture-of-Experts (MoE) models, a problem stemming from the neglect of heterogeneous routing structures established during pretraining. The authors propose RA-MoE, a novel framework that reveals, for the first time, a strong correlation between routing discrepancies in intermediate layers and multilingual performance gaps. Leveraging a language-agnostic alignment region, RA-MoE categorizes samples into four types based on prediction correctness in English versus the target language to identify task-relevant experts. It then introduces a routing alignment loss that encourages the target language to mimic English expert activation patterns on challenging samples. Evaluated across three MoE architectures, three tasks, and six languages, the proposed three-stage fine-tuning strategy consistently outperforms standard supervised fine-tuning and strong baselines such as Routing Steering and RISE, with alignment gains effectively predictable by the proportion of difficult samples.

Abstract

Mixture-of-Experts (MoE) models have emerged as a dominant paradigm for efficient LLM scaling, yet adapting them to non-English downstream tasks remains challenging. Existing fine-tuning approaches treat MoE models as monolithic learners, ignoring the heterogeneous routing structure that develops during pretraining. We validate across multiple MoE models and downstream tasks that middle layers form a language-universal alignment zone where routing divergence strongly predicts per-language task performance gaps. Building on this observation, we propose RA-MoE (Routing-Aligned MoE Fine-Tuning), a three-stage framework that categorizes parallel task examples into a four-way taxonomy (cc/ci/ic/ii) based on correctness in English and the target language, identifies task-relevant experts in the middle layers, and augments standard SFT with a routing alignment loss that encourages target-language routing on ci-type examples to follow the English task-expert activation pattern. Experiments across three MoE models, three tasks, and six target languages demonstrate that RA-MoE consistently outperforms standard SFT and strong baselines including Routing Steering and RISE, with the ci proportion of a task-language pair serving as a reliable predictor of alignment benefit.

Problem

• Mixture-of-Experts
• multilingual adaptation
• routing alignment
• fine-tuning
• language performance gap

Innovation

• Mixture-of-Experts
• routing alignment
• multilingual fine-tuning
• language-universal zone
• expert routing