Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution

2026-05-08

AI Summary

This work addresses the challenge of achieving fine-grained sparsity in Mixture-of-Experts (MoE) models, which often leads to training instability and load imbalance despite their inherent efficiency from sparse expert activation. The study systematically uncovers and leverages a previously overlooked yet prevalent form of intra-expert activation sparsity present in off-the-shelf MoE models. Without modifying model architecture or requiring retraining, the method dynamically skips computation for inactive neurons during inference. By integrating this sparsity into the vLLM inference pipeline alongside existing optimizations, the approach significantly enhances computational efficiency. Evaluated across eight MoE models ranging from 1B to 400B parameters, it achieves up to 2.5× speedup in MoE layers and 1.2× end-to-end inference acceleration with negligible accuracy degradation.

Abstract

Mixture of Experts (MoE) architecture has become the standard for state-of-the-art large language models, owing to its computational efficiency through sparse expert activation. However, sparsity through finer expert granularity is becoming increasingly difficult to achieve due to fundamental training challenges such as expert collapse and load imbalance. In this work, we explore and leverage intra-expert activation sparsity as a complementary and underexplored dimension of sparsity in MoE models. Surprisingly, substantial intra-expert sparsity is readily available in existing pre-trained MoE models, without any modification to the activation function or model parameters, providing up to 90% sparsity within each expert without significant accuracy loss. We explore intra-expert activation sparsity across eight off-the-shelf MoE models ranging from 1B to 400B parameters, and extend the MoE execution pipeline of vLLM to leverage intra-expert activation sparsity by skipping the computations of inactive neurons, on top of its existing optimizations, achieving up to 2.5 times speedup in MoE layer execution and 1.2 times end-to-end speedup compared to the original dense vLLM baseline.

Problem

• Mixture-of-Experts
• activation sparsity
• expert collapse
• load imbalance
• model efficiency

Innovation

• intra-expert sparsity
• Mixture-of-Experts
• activation sparsity
• efficient inference
• vLLM