FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

2026-06-17

AI Summary

This work addresses the high memory and communication overheads in cross-datacenter large language model (LLM) pretraining, where each site traditionally maintains a full model replica. To overcome this limitation, the authors propose an efficient MoE-based distributed pretraining approach that partitions expert layers across nodes, selectively replicates a subset of experts, incorporates a skip-token mechanism, and employs a low-communication data-parallel strategy. Notably, this is the first method to integrate partial expert replication into federated-style training, substantially reducing both communication and memory costs while ensuring routing stability. Experimental results demonstrate that the proposed method reduces communication overhead by 1.42× and 45.44× compared to strong baselines and standard DDP, respectively, achieves up to 1.4× higher throughput, and scales effectively to models with hundreds of billions of parameters.

Abstract

Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators. While increasing model and dataset scale remains the dominant driver of performance, Mixture-of-Experts (MoEs) architectures have recently achieved state-of-the-art results by decoupling parameter count from computational cost. This efficiency enables training massive models on constrained compute budgets, yet it typically requires the high-speed interconnects of a single datacenter. To overcome these physical limits, recent approaches such as DiLoCo and Photon use low-communication data-parallel methods to enable scaling across geographically distributed, weakly connected data centers. However, these methods suffer from a fundamental inefficiency: they require full model replicas at every site, which imposes prohibitive memory constraints and communication overheads. In this work, we introduce FoMoE, a system that breaks the full-replica paradigm by partitioning expert layers across workers. We demonstrate that FoMoE: (I) reduces communication costs by up to 1.42x over efficient baselines and 45.44x over DDP via partial expert replication in the studied regimes; (II) achieves empirical throughput speedups of up to 1.4x through a novel skip-token mechanism; and (III) shows stable routing in the trained proxy regimes and projects the communication/memory benefits to 100B-scale configurations through system modelling.

Problem

• Federated Learning
• Mixture-of-Experts
• Model Replication
• Distributed Training
• Large Language Models

Innovation

• Mixture-of-Experts
• distributed training
• model partitioning
• skip-token mechanism
• full-replica elimination