FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

2026-01-29

AI Summary

This work addresses the challenge that current large language models struggle to align with user intent during pretraining due to insufficient supervised instruction data. To overcome this limitation, we introduce FineInstructions, a large-scale synthetic dataset comprising billions of high-quality instruction–response pairs, automatically generated by matching internet-scale unstructured corpora with approximately 18 million instruction templates derived from real user queries. Leveraging this dataset, we present the first approach to pretrain a language model from scratch using purely instruction-tuning objectives, thereby departing from conventional self-supervised paradigms. Experimental results demonstrate that, at equal token budgets, our method significantly outperforms standard pretraining and alternative synthetic data strategies, achieving superior response quality on standard benchmarks for open-ended generation tasks.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised"predict the next word"objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of"instruction-tuning"data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With"supervised"synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .

Problem

• instruction tuning
• supervised training data
• large language models
• synthetic data
• pre-training

Innovation

• instruction tuning
• synthetic data generation
• large language models
• pre-training
• template-based instantiation