Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression

2026-06-16

AI Summary

This work addresses the challenge of deploying Mixture-of-Experts (MoE) models, which suffer from high memory consumption and inference overhead. Existing compression methods apply coarse-grained pruning at the expert level, overlooking fine-grained redundancy within experts. To overcome this limitation, the paper proposes the first channel-level structured pruning framework for MoE models. It leverages attribution analysis to identify channels where information is concentrated and formulates the allocation of pruning ratios as a channel-score coverage maximization problem, which is efficiently solved to derive an optimal pruning strategy. Combined with 4-bit quantization, the method achieves nearly lossless accuracy under 50% or 25% structured pruning on DeepSeek and Qwen MoE models, respectively, and reduces memory usage by up to 5.27× on Qwen3-30B-A3B, significantly outperforming current state-of-the-art approaches.

Abstract

Mixture-of-Experts (MoE) models scale compute efficiently, yet remain expensive to deploy due to their substantial memory footprint and inference overhead. Prior compression methods mainly operate at the expert level, either removing entire experts or ranking experts by coarse-grained importance scores. However, such expert-wise decisions are often too coarse to capture fine-grained redundancy, leading to misallocated pruning budgets and limited compression. To address this problem, we observe that information within MoE experts is highly concentrated in a small subset of channels, leaving substantial redundancy even in experts deemed important. Based on this observation, we propose a structural pruning framework tailored for MoE models. Our method reformulates prune-ratio allocation as a channel-score coverage maximization problem and solves it efficiently using an attribution-based approximation. Experiments on DeepSeek and Qwen MoE models show that our method preserves model accuracy under 50% or 25% structured pruning when combined with 4-bit quantization. On Qwen3-30B-A3B, our approach reduces memory footprint by 5.27$\times$ and consistently outperforms state-of-the-art baselines across diverse benchmarks.

Problem

• Mixture-of-Experts
• model compression
• structural pruning
• redundancy
• memory footprint

Innovation

• structural pruning
• Mixture-of-Experts
• attribution-guided
• coverage maximization
• channel-level compression