Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

2026-08-11

AI Summary

This work addresses the disconnect among algorithmic design, model architecture, and system implementation in conventional large-model pretraining, which often prevents computationally optimal solutions from satisfying real-world cluster efficiency constraints. To bridge this gap, the authors propose MOSAIC, a framework that jointly models the architecture and system implementation of sparse Mixture-of-Experts (MoE) models. By integrating system-aware constraints—including Model FLOPs Utilization (MFU), communication overhead, memory footprint, and parallelization strategies—into predictive scaling laws, MOSAIC reveals the inconsistency between computation-optimal and cluster-optimal configurations and derives the sparsity setting that maximizes performance under practical system constraints. Experiments across models with up to 79 billion total parameters and activated parameters ranging from 104 million to 2.7 billion demonstrate that MOSAIC accurately identifies sparsity configurations balancing loss and hardware efficiency, significantly outperforming baselines that ignore system-level factors.

Abstract

In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages. A scaling law stage selects an architecture and training recipe, optimizing loss under compute constraints, and a separate systems stage then optimizes the implementation for hardware efficiency. In this work, we develop MOSAIC, which formulates model architecture and systems co-design as an optimization problem. MOSAIC couples a predictive scaling law with a calibrated performance model that estimates Model FLOPs Utilization (MFU), communication cost, memory footprint, and the best parallel layout. We instantiate the framework for sparse Mixture-of-Experts (MoE) language models, where expert count, routing sparsity, and other MoE layer dimensions affect both the loss and systems efficiency. We fit a scaling law on sparse MoE models trained on text data, whose scaling dimensions include the sparsity factor, which is the fraction of model parameters inactive per token in a forward pass. The scaling law sweeps in our work span active parameters from $104$ million to $2.7$ billion and total model sizes reaching $79$ billion parameters. We show that, within the calibrated sparsity range, an efficiency-agnostic model-FLOPs budget admits no interior optimal sparsity. The fitted loss decreases monotonically with sparser models and the compute optimum lies at the upper boundary of the data support. An optimal sparsity in MoE models instead emerges under the cluster's systems constraints, as captured by MOSAIC. Our results argue for a shift towards unified architecture and systems co-design for frontier language model training.

Problem

• Mixture-of-Experts
• scaling laws
• systems co-design
• sparsity
• large language models

Innovation

• Mixture-of-Experts
• systems-aware scaling
• model-system co-design
• sparsity optimization
• scaling laws