Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration

2026-08-28

AI Summary

研究通过采用异构层模式，集中专家容量并增加额外的token-mixing和dense-FFN层，以减少训练成本并提高效率。

Abstract

When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth. Compared to conventional models which interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additional token-mixing and dense-FFN layers. Across a scaling ladder from 2B to 31.5B total parameters, under matched total and activated parameters, CE-MoE models consistently reduce training cost while matching validation loss and downstream benchmarks with full-MoE baselines. At the 31.5B scale, CE-MoE uses 33.3\% fewer GPU-hours while improving average downstream score and inference throughput.

Problem

• Mixture-of-Experts
• communication-efficient
• token dispatch
• combine collectives
• training cost

Innovation

• Communication-Efficient
• Heterogeneous Layer Pattern
• Token-Mixing and Channel-Mixing Decoupling
• Training Cost Reduction