Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages

2026-08-09

AI Summary

This study investigates whether pre-pretraining—pretraining large language models on artificial languages—consistently enhances training efficiency in multilingual settings. The authors conduct a systematic evaluation across multiple natural languages spanning four major language families, varying tokenizers (including Llama), model scales, and using 128-Dyck as the artificial pre-pretraining language. Rigorous experimentation includes multiple random seeds and linguistic quantitative metrics such as sentence length, morphological complexity, and dependency tree features. The findings reveal, for the first time, that the average 33% efficiency gain from pre-pretraining is highly sensitive to experimental configurations and stochasticity. Stable improvements are observed predominantly with smaller models combined with the Llama tokenizer across most languages, thereby establishing a quantitative link between performance gains and structural linguistic properties.

Abstract

Pretraining LLMs on artificial languages ("pre-pretraining") is a technique that could reportedly increase token efficiency by 33%, i.e., save up to 33% of training tokens needed to reach a certain performance. We validate this prior result for English on a larger set of natural languages across four language families, using two different tokenizers and varying model sizes. We also relate the observed gains (or losses) in token efficiency to quantified linguistic properties of the languages, such as sentence length, morphological richness, and features of dependency syntactic trees (tree depth, number of children, number of crossing dependencies). Our empirical results indicate that the reported gains depend heavily on the experiment setup and the choice of random seed, although we can confirm the trend of stable gains with 128-Dyck pretraining of small models with the Llama tokenizer for most of the examined languages. On a general note, we argue that multiple training runs should be carried out at least for a subset of experiments to avoid the community adopting unstable approaches.

Problem

• pre-pretraining
• token efficiency
• language stability
• large language models
• cross-lingual generalization

Innovation

• pre-pretraining
• token efficiency
• linguistic properties
• instability
• multilingual evaluation