Efficiently Editing Mixture-of-Experts Models with Compressed Experts

2025-03-01

AI Summary

To address the high inference overhead caused by expert redundancy in fine-tuning Mixture-of-Experts (MoE) models, this paper proposes the “Compressed Expert” mechanism: dynamically identifying low-contribution experts via importance scoring and replacing them with lightweight modules—while preserving high-contribution experts intact—enabling expert-granularity, parameter-selective compression. The method jointly integrates expert-level knowledge distillation and structured parameter compression, avoiding full-model pruning. Experiments on Phi-MoE and OLMoE demonstrate that our approach recovers over 90% of the full-expert performance, reduces activated parameters by more than 30%, and lowers inference latency and computational cost by approximately 20%, significantly improving deployment efficiency under resource constraints. This work introduces, for the first time, dynamic, expert-level compression into MoE fine-tuning, establishing a novel paradigm for efficient MoE deployment.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Data Mining & Knowledge Management: Data Compression
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Mixture-of-Experts (MoE) models have become a key approach for scaling large language models efficiently by activating only a subset of experts during training and inference. Typically, the number of activated experts presents a trade-off: fewer experts reduce computational costs, while more experts improve performance. Recent studies reveal that not all activated experts contribute equally to model performance, with some providing minimal utility, particularly when finetuning pretrained MoE models for specialized downstream tasks. The co-existence of significant and redundant parameters in experts provides us an opportunity to reduce the number of activated experts while maintaining model performance. In this work, we propose the concept of compressed experts, lightweight modules that serve as compact representations of full experts. Our approach preserves the most important experts while replacing other auxiliary activated experts with compressed experts. The reduction of active parameters significantly lowers inference costs while achieving comparable performance. Extensive experiments on models including Phi-MoE and OLMoE demonstrate that compressed experts recover over 90% of full expert performance across various tasks while reducing more than 30% active parameters and saving 20% in inference costs. This approach enables efficient deployment of MoE models in resource-constrained settings and facilitates scaling to larger models with manageable overhead. Our code is available at https://github.com/yifei-he/Compressed-Experts.

Problem

• Reduces computational costs in Mixture-of-Experts models.
• Maintains model performance with fewer activated experts.
• Enables efficient deployment in resource-constrained settings.

Innovation

• Compressed experts reduce active parameters significantly.
• Lightweight modules replace auxiliary experts efficiently.
• Maintains performance while lowering inference costs.