UMoE:Unlocking Every Expert in Domain-Specific Training

2026-07-13

AI Summary

This work addresses the inefficacy of standard supervised fine-tuning (SFT) in domain-specific adaptation of mixture-of-experts (MoE) models, where pre-trained expert pools often contain numerous experts with negligible relevance to the target domain. The authors propose a three-stage method under a fixed computational budget: first pruning weakly aligned experts based on domain alignment saliency, then restoring expert capacity via perturbation-driven expansion, and finally applying standard SFT. This approach uniquely reconfigures the expert pool to better suit the target domain without altering model size or inference cost, and without requiring domain-specific hyperparameter tuning. Evaluated across two MoE architectures, five domains, and twelve benchmarks, the method consistently outperforms direct SFT, yielding an average accuracy gain of 3.4 points, a 6.0-point improvement on SWE-bench Verified, and a 1.36-point advantage even under strong SFT baselines.

Abstract

Mixture-of-Experts (MoE) models scale capacity without proportional compute cost and have become a key architecture for frontier large language models (LLMs). Yet domain-specific post-training inherits an expert pool shaped by mixed-domain pre-training: a substantial subset of experts contributes little on the target domain, and standard supervised fine-tuning (SFT) leaves the composition of this pool unchanged. We propose a simple, budget-preserving pipeline that realigns the expert pool to the target domain before fine-tuning. Given a target domain, we (1) prune the experts with lowest domain-aligned saliency, (2) regrow the expert pool to its original size through perturbation-based expert expansion, and (3) apply standard SFT. The resulting model preserves the original expert count, parameter count, and inference cost. With a single frozen recipe and no per-domain hyperparameter tuning, UMoE consistently improves over direct sft across two MoE architectures (Qwen3-30B-A3B and Qwen3.5-35B-A3B), five domains (math, code, science, tool-use, and agentic coding), and 12 benchmarks. Representative improvements are 3.4 points in math average accuracy, 6.0 points on SWE-bench Verified. On a strong in-house math corpus, direct sft already surpasses Qwen3-30B-A3B-Thinking (82.81 vs.\ 81.06), yet UMoE further raises the average to 84.17, an additional 1.36 points, demonstrating robustness to a substantially stronger SFT regime. Data-scaling experiments further show that the gain persists as training data grows. Analysis reveals that the direct-SFT model allocates substantial routed-expert compute to a low-saliency subset that can be removed post hoc with little average degradation; UMoE turns this redundant capacity into useful domain capacity and achieves lower training loss, with gains spanning all difficulty levels in downstream evaluation.

Problem

• Mixture-of-Experts
• domain-specific training
• expert pruning
• supervised fine-tuning
• expert realignment

Innovation

• Mixture-of-Experts
• domain-specific adaptation
• expert pruning and regrowth
• parameter-efficient fine-tuning
• saliency-based realignment