Generating Pretraining Tokens from Organic Data for Data-Bound Scaling

2026-05-18

AI Summary

This work addresses the looming bottleneck of organic text data exhaustion in large language model pretraining, where existing approaches struggle to fully exploit the learning potential of limited datasets. The authors propose SynPro, a novel framework that enables model-aware, faithful synthetic data generation by performing fidelity-preserving paraphrasing and format restructuring—without introducing external information—to expand high-quality pretraining tokens. SynPro dynamically optimizes its generation strategy via reinforcement learning, guided by a reward mechanism incorporating quality, faithfulness, and data influence. Using only 10% of the Chinchilla-optimal token budget, SynPro yields 3.7–5.2 times more effective learning tokens than naive repetition, and its 1.1B-parameter model even surpasses an ideal baseline trained on an equivalent amount of unique real data, substantially mitigating distributional collapse under data scarcity.

Abstract

LLM pretraining is shifting from a compute-bound to a data-bound regime, where available human (organic) text falls far short of scaling demands. However, reaching the data-bound regime does not mean the model has fully utilized its organic corpus. In this paper, we introduce SynPro, a synthetic data generation framework that helps LLMs more thoroughly learn from limited organic data. SynPro applies two operations, rephrasing and reformat, that present the same organic source in diverse forms to facilitate deeper learning without introducing external information. Both generators are optimized via reinforcement learning with quality, faithfulness, and data influence rewards, and are continuously updated as pretraining plateaus to target content the model has yet to absorb. We pretrain 400M and 1.1B models with 10% of their Chinchilla-optimal tokens (0.8B and 2.2B) from DCLM-Baseline, reflecting a realistic data-bound regime in frontier pretraining. Our results reveal that organic data is significantly underutilized by standard repetition: SynPro unlocks 3.7-5.2x the effective tokens of repetition, even surpassing the non-data-bound oracle that trains on equivalent unique data at the 1.1B scale. Analyses confirm that faithful, model-aware synthesis sustains data-bound scaling without causing distribution collapse. We open-source our code at https://github.com/cxcscmu/SynPro.

Problem

• data-bound scaling
• organic data
• LLM pretraining
• data underutilization
• synthetic data

Innovation

• synthetic data generation
• data-bound scaling
• reinforcement learning
• faithful rephrasing
• model-aware synthesis