DeaMoE: Efficient MoE Structure for Fast Small-Batch Decoding

2026-08-14

AI Summary

This study addresses the memory bottleneck caused by expert weight loading in Mixture-of-Experts (MoE) models during small-batch decoding. We propose DeaMoE, a novel architecture that integrates expert group sharing with private parameters and employs a two-stage routing strategy to eliminate redundant weight loading while preserving model expressiveness and inference efficiency. Experimental results demonstrate that DeaMoE reduces per-step weight loading by 50.9% compared to conventional MoE. Furthermore, it achieves a 1.33× speedup in Time Per Output Token (TPOT) for a 7B model on NVIDIA A40 GPUs, and peak speedups of 2.00× and 1.97× for DeepSeek-V3 on A40 and H100 GPUs, respectively. These improvements significantly optimize inference latency for real-time interactive applications.

Abstract

Mixture-of-Experts (MoE) models have been widely adopted in real-time interactive applications such as coding assistants, real-time audio-video interaction systems. To meet the extremely low response latency requirements of these scenarios, practitioners commonly employ small-batch decoding, under which MoE inference becomes memory-bound and is severely bottlenecked by expert weight loading. However, this bottleneck has received limited attention, and existing solutions such as post-training weight compression or fine-grained expert design during pre-training either degrade model accuracy or introduce additional computation and communication overhead. To tackle this issue, we propose DeaMoE, a decoding-efficient MoE architecture, in which the experts are grouped into several departments, and the experts belonging to the same department share most parameters since they come from the same professional field, and additionally each expert contains a few private parameters to reflect its uniqueness. Moreover, we design customized two-stage routing strategy for DeaMoE to avoid redundant loading, under which DeaMoE greatly improves the efficiency during LLM decoding. Compared with vanilla MoE, DeaMoE reduces per-step loaded weights by up to 50.9% and achieves up to 1.33 end-to-end TPOT speedup for the pre-trained 7B model on A40, and up to 2.00x and 1.97x peak speedup for DeepSeek-V3 on A40 and H100 in microbenchmarks.

Problem

• Mixture-of-Experts
• Small-batch Decoding
• Memory-bound
• Expert Weight Loading
• Inference Latency

Innovation

• Mixture-of-Experts
• Small-Batch Decoding
• Parameter Sharing
• Two-Stage Routing
• Memory-Bound