Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

2026-04-21

AI Summary

This work addresses the challenge of efficiently scaling large Mixture-of-Experts (MoE) models, whose memory and communication overhead grow prohibitively with parameter count. The authors propose “expert upcycling,” a method that incrementally increases the number of experts during continued pretraining by duplicating existing experts and expanding the router, while keeping per-token computation constant. The approach integrates a warm-start initialization, a utility-based non-uniform expert duplication strategy, and a gradient-importance-driven expert selection mechanism. A theoretical framework is provided to analyze the resulting quality gap. Experiments on models ranging from 7B to 13B parameters demonstrate that upcycled models match the validation loss of baselines while reducing GPU training time by 32%, substantially improving scaling efficiency and model extensibility.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Mixture-of-Experts (MoE) has become the dominant architecture for scaling large language models: frontier models routinely decouple total parameters from per-token computation through sparse expert routing. Scaling laws show that under fixed active computation, model quality scales predictably with total parameters, and MoEs realize this by increasing expert count. However, training large MoEs is expensive, as memory requirements and inter-device communication both scale with total parameter count. We propose expert upcycling, a method for progressively expanding MoE capacity by increasing the number of experts during continued pre-training (CPT). Given a trained E-expert model, the upcycling operator constructs an mE-expert model through expert duplication and router extension while holding top-K routing fixed, preserving per-token inference cost. Duplication provides a warm initialization: the expanded model inherits the source checkpoint's learned representations, starting from a substantially lower loss than random initialization. Subsequent CPT then breaks the symmetry among duplicated experts to drive specialization. We formalize the upcycling operator and develop a theoretical framework decomposing the quality gap into a capacity term and an initialization term. We further introduce utility-based expert selection, which uses gradient-based importance scores to guide non-uniform duplication, more than tripling gap closure when CPT is limited. In our 7B-13B total parameter experiments, the upcycled model matches the fixed-size baseline on validation loss while saving 32% of GPU hours. Comprehensive ablations across model scales, activation ratios, MoE architectures, and training budgets yield a practical recipe for deploying expert upcycling, establishing it as a principled, compute-efficient alternative to training large MoE models from scratch.

Problem

• Mixture-of-Experts
• compute efficiency
• model scaling
• training cost
• sparse routing

Innovation

• expert upcycling
• Mixture-of-Experts
• compute-efficient scaling
• continued pre-training
• expert duplication