ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compression

2026-05-28

AI Summary

This work addresses the high memory overhead of deploying Mixture-of-Experts (MoE) language models, which stems from the need to load all experts during inference. The authors frame MoE compression as a problem of expert pool consolidation and propose a fine-tuning-free prototype reallocation framework. By leveraging calibration data to assess each expert’s contribution and replaceability, the method selects a small subset of pretrained experts as prototypes and deterministically remaps original experts onto these prototypes. This approach preserves the original routing interface while enabling intra-layer prototype sharing. Evaluated on three models—including DeepSeek-MoE-16B—the method consistently matches or outperforms existing pruning and merging strategies, achieving state-of-the-art average performance at both 25% and 50% expert reduction rates.

Abstract

Mixture-of-Experts (MoE) language models reduce per-token computation but still require storing and serving all experts, making deployment memory-intensive. Existing post-training compression methods mainly shrink this cost by pruning experts or merging their weights. We formulate post-training MoE compression as expert-pool consolidation: retaining a smaller set of pretrained experts as reusable prototypes and deterministically remapping each original expert reference to one selected prototype. This view separates the reduced expert pool from the reuse structure that represents the original expert slots, and allows prototype sharing within local layer scopes while preserving the original router interface. We propose ConMoE, a train-free prototype remapping framework that selects retained experts using calibration-based contribution and replaceability signals, then redirects original expert calls to the selected prototypes without weight updates or post-compression fine-tuning. Experiments on three pretrained MoE language models show that ConMoE matches or outperforms strong pruning and merging baselines in several settings, achieving the best average score on deepseek-moe-16b-base at both 25% and 50% routed-expert reduction, while remaining competitive on Qwen3-30B-A3B and OLMoE-1B-7B-0125. Ablations indicate that deterministic reassignment is the most stable component, whereas broader cross-layer sharing and post-hoc weight fusion are model-dependent.

Problem

• Mixture-of-Experts
• model compression
• post-training compression
• expert pruning
• memory efficiency

Innovation

• Mixture-of-Experts
• model compression
• prototype remapping
• expert consolidation
• post-training compression