Metadata Conditioning Accelerates Language Model Pre-training

2025-01-03

AI Summary

To address low learning efficiency and poor deployment flexibility in language model pretraining on heterogeneous data, this paper proposes MeCo (Metadata-Conditioned Pretraining). MeCo injects metadata—such as URLs—as conditional signals during initial pretraining, then smoothly transitions to metadata-free training via a novel metadata annealing phase. This introduces the first two-stage paradigm combining metadata-conditioned modeling with progressive annealing. The method enables controllable generation guided by either real or synthetic metadata, incurs zero computational overhead, and generalizes across model scales and diverse data sources (e.g., C4, RefinedWeb, DCLM). Experiments show that a 1.6B-parameter model achieves comparable performance using only 67% of the original pretraining data, with consistent training speedup across all benchmarks. Moreover, MeCo significantly reduces harmful outputs and improves accuracy on commonsense reasoning tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Metareasoning and Metaheuristics
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The vast diversity of styles, domains, and quality levels present in language model pre-training corpora is essential in developing general model capabilities, but efficiently learning and deploying the correct behaviors exemplified in each of these heterogeneous data sources is challenging. To address this, we propose a new method, termed Metadata Conditioning then Cooldown (MeCo), to incorporate additional learning cues during pre-training. MeCo first provides metadata (e.g., URLs like en.wikipedia.org) alongside the text during training and later uses a cooldown phase with only the standard text, thereby enabling the model to function normally even without metadata. MeCo significantly accelerates pre-training across different model scales (600M to 8B parameters) and training sources (C4, RefinedWeb, and DCLM). For instance, a 1.6B language model trained with MeCo matches the downstream task performance of standard pre-training while using 33% less data. Additionally, MeCo enables us to steer language models by conditioning the inference prompt on either real or fabricated metadata that encodes the desired properties of the output: for example, prepending wikipedia.org to reduce harmful generations or factquizmaster.com (fabricated) to improve common knowledge task performance. We also demonstrate that MeCo is compatible with different types of metadata, such as model-generated topics. MeCo is remarkably simple, adds no computational overhead, and demonstrates promise in producing more capable and steerable language models.

Problem

• Language Model Pretraining
• Efficiency
• Diverse Data

Innovation

• Metadata Conditioning
• Language Model Efficiency
• Controllable Text Generation