It Takes a MAESTRO To Prune Bad Experts

2026-07-09

AI Summary

This work addresses the deployment bottleneck of sparsely activated Mixture-of-Experts (MoE) models, which retain all experts in memory despite activating only a subset during inference. Existing pruning methods often overlook routing dependencies across layers, limiting their compression efficacy. To overcome this, the authors propose MAESTRO, a novel framework that models expert activation trajectories as ergodic Markov chains and leverages their stationary distributions to capture global routing dependencies for structured pruning. By moving beyond local heuristics, MAESTRO enables routing-consistent importance estimation, significantly improving generalization consistency after compression. Under a 50% compression ratio, the pruned models achieve an average performance retention gain of 10.61% across five domains—including safety, bias, and ethics—and exhibit substantially reduced cross-task performance variance.

Abstract

Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.

Problem

• Mixture-of-Experts
• structured pruning
• expert sparsification
• routing dependencies
• deployment bottleneck

Innovation

• Mixture-of-Experts
• structured pruning
• Markov chain
• expert sparsification
• routing-aware