Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model

2026-08-13

AI Summary

This work investigates whether pretraining of large language models can be decomposed into independent smaller-scale tasks and later reassembled into a fully functional model. To this end, the authors propose Mixture of Training (MoT), a method that partitions a target Transformer into contiguous layer blocks, trains these blocks in parallel within a frozen pretrained aligner scaffold, and subsequently reassembles them followed by brief end-to-end fine-tuning. Evaluated on the Gemma architecture using the C4 dataset, MoT achieves perplexity comparable to monolithic training on a 1.3B-parameter model despite processing more total tokens, and demonstrates potential computational efficiency when reusing the aligner. This study provides the first empirical validation that deep architectural slices can be independently trained and effectively recombined into a coherent large language model.

Abstract

We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent larger model. We introduce Mixture of Training (MoT), a scaffolded modular pre-training procedure that partitions a target Transformer into contiguous layer blocks, trains each block inside a frozen pretrained aligner scaffold, and then recomposes the trained blocks with an optional short end-to-end adaptation pass. On a 1.3B-parameter Gemma-style model trained on C4, MoT provides a small-scale proof of mechanism: independently trained depth slices can be recomposed into a usable language model, and a quality-parity schedule reaches the same reported perplexity as the monolithic baseline. This parity setting processes more aggregate tokens and has a shorter idealized layer-equivalent critical path after aligner preparation; its effective compute advantage depends on reusing the aligner across runs. We therefore present MoT not as a general replacement for monolithic pre-training, but as a small-scale framework for studying whether scaffolded sub-runs can act as reusable training units.

Problem

• language model pretraining
• modular training
• model decomposition
• scaffolded pretraining
• training scalability

Innovation

• Mixture of Training
• modular pretraining
• scaffolded training
• layer-wise decomposition
• recomposable language models