Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models

2026-07-02

AI Summary

This work addresses the limitation of existing Mixture-of-Experts (MoE) pruning methods, which rely on a single importance score and thus tend to favor experts specialized on specific corpora when downstream calibration data is unavailable, thereby compromising model generalization. To mitigate this bias, the authors propose Generic TB-Coverage, a pruning strategy that leverages only generic corpora such as WikiText2 and C4. By constructing utility profiles of each expert across multiple corpora and applying a fixed-budget coverage rule, the method prioritizes experts demonstrating consistently high utility across diverse data sources to generate pruning masks. Evaluated on Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base, Generic TB-Coverage significantly outperforms REAP, ExpertSparsity, and random pruning across retention rates of 25%–75%, achieving higher average zero-shot accuracy on six benchmarks and incurring smaller perplexity degradation on generic corpora.

Abstract

Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set toward experts favored by dominant calibration patterns. We propose \textbf{Generic TB-Coverage}, a coverage-aware expert pruning method that uses only generic text corpora (WikiText2 and C4) for calibration. Instead of collapsing expert utility into one score, our method profiles per-expert utility separately on each corpus and enforces a fixed-budget coverage rule that preserves high-utility experts from each corpus before constructing the final pruning mask. Across Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base at 25\%, 50\%, and 75\% retention budgets, our method improves average accuracy on six common zero-shot benchmarks over random pruning, REAP, and ExpertSparsity, while also reducing perplexity degradation on WikiText2 and C4. The gains are largest under aggressive pruning (25\% and 50\% retain), suggesting that preserving cross-corpus expert coverage is an effective generic-data prior for MoE pruning. Our improvements hold with fixed pruning budgets and no downstream calibration data.

Problem

• Mixture-of-Experts
• expert pruning
• calibration-free
• structured redundancy
• sparse language models

Innovation

• Mixture-of-Experts
• expert pruning
• coverage-aware
• generic calibration
• structured redundancy