SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling

2026-06-15

AI Summary

This work addresses the challenge in data-constrained supervised fine-tuning of Mixture-of-Experts (MoE) models, where existing adaptation methods often induce expert homogenization or excessively perturb pretrained parameters, failing to balance architectural reuse with diversity. To overcome this, we propose SVD-Partitioned Residual Initialization (SPRI), which— for the first time— decomposes the pretrained feed-forward network (FFN) weights via singular value decomposition (SVD) and allocates the resulting residual components across distinct experts. This approach preserves the spectral structure of the original weights while introducing controlled diversity, further enhanced by a two-stage training strategy to improve stability. Evaluated on 15 English-to-many translation directions in CoVoST2, SPRI achieves average gains of 2.58 BLEU and 3.32 COMET points over fully fine-tuned dense models, and outperforms the previous best MoE method by 3.39 BLEU and 4.34 COMET points.

Abstract

Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive. MoE upcycling mitigates this cost by converting pretrained dense models into sparse MoE models. However, existing upcycling methods typically rely on large-scale continued training and often perform poorly under data-constrained supervised adaptation, due to either homogeneous experts or overly disruptive perturbations to pretrained parameters. In this setting, effective upcycling must leverage pretrained weight structure while introducing sufficient diversity among routed experts. To this end, we propose SVD-Partitioned Residual Initialization (SPRI), which distributes SVD-partitioned residuals derived from pretrained feed-forward network (FFN) weights across routed experts, introducing controlled expert diversity grounded in pretrained spectral structure. We further introduce a two-stage training strategy to improve adaptation stability. We evaluate SPRI on multilingual speech-to-text translation, where limited supervised data challenges MoE upcycling and multiple target languages provide natural routing heterogeneity. On CoVoST2 across 15 En-to-XX directions, SPRI improves average BLEU and COMET over fully fine-tuned dense models by 2.58 and 3.32 points, respectively, and outperforms the prior best MoE upcycling baseline by 3.39 BLEU and 4.34 COMET points.

Problem

• Mixture-of-Experts
• MoE upcycling
• data-constrained adaptation
• expert diversity
• pretrained models

Innovation

• Mixture-of-Experts
• SVD-Partitioned Residual Initialization
• MoE upcycling
• data-constrained adaptation
• expert diversity