How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle

2026-06-14

AI Summary

Existing one-shot expert pruning methods lack a unified principle, making them difficult to adapt to diverse deployment objectives. This work proposes the first unified one-shot pruning framework tailored for Mixture-of-Experts (MoE) models, integrating three key factors—routing frequency, gating weights, and activation magnitude—to systematically establish principled guidelines for selecting pruning criteria in both task-agnostic and task-specific settings. Building on this framework, we introduce two novel task-agnostic criteria, MAN and MSAN, along with an expert scoring mechanism based on activation norms. Extensive experiments across four MoE models and sixteen benchmarks demonstrate that MAN and MSAN consistently rank among the top two performers in task-agnostic scenarios, achieving performance gains of up to 8.8 points.

Abstract

Mixture-of-Experts (MoE) language models reduce per-token computation through sparse expert activation, yet deployment still requires storing the full expert pool, making one-shot expert pruning a practical approach for reducing memory usage. Although effective, existing criteria are largely heuristic, and no single criterion is universally optimal. Thus, establishing a principle for selecting pruning criteria suited to different deployment objectives remains an important yet largely underexplored problem in one-shot expert pruning. To this end, we introduce a unified formulation for one-shot MoE expert pruning organized around three factors: routing frequency, gate weighting, and activation strength. The formulation yields a criteria selection principle: task-agnostic pruning should favor routed-token-averaged, gate-free activation-based criteria, whereas task-specific pruning can benefit from retaining routing-frequency and gate-weight information. Beyond this principle, the formulation also provides a systematic view of existing heuristic criteria and gives rise to two new task-agnostic criteria, Mean Activation Norm (MAN) and Mean Squared Activation Norm (MSAN). Across four representative MoE models and 16 diverse benchmarks, MAN and MSAN are consistently strong in the task-agnostic setting, obtain the top-two average ranks, and improve average performance by up to 8.8 points over the strongest baseline.

Problem

• one-shot expert pruning
• Mixture-of-Experts
• pruning criteria
• MoE language models
• expert selection

Innovation

• Mixture-of-Experts
• one-shot pruning
• expert scoring
• activation norm
• unified formulation