How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

2026-04-15

AI Summary

This work addresses the lack of systematic optimization in existing synthetic pretraining data generation, particularly concerning prompt design, generator models, and source data selection. Through large-scale controlled experiments, we systematically investigate how to efficiently rewrite web text into high-quality synthetic data, with a focus on the impact of structured output formats (e.g., tables, math problems, FAQs), generator scale, and source data choices. Our findings reveal that structured formats substantially outperform current approaches, and that generator performance saturates beyond 1B parameters. Leveraging these insights, we propose a cost-effective synthesis strategy. Based on trillion-token-scale experiments, we release an open-source generation framework and the FinePhrase dataset—comprising 486 billion tokens—that surpasses all existing synthetic baselines in performance while reducing generation costs by up to 30×.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop \textbf{\textsc{FinePhrase}}, a 486-billion-token open dataset of rephrased web text. We show that \textsc{FinePhrase} outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community.

Problem

• synthetic data
• pretraining data
• prompt design
• generator model
• source data

Innovation

• synthetic pretraining data
• prompt design
• structured output formats
• generator model scaling
• source data selection