When Bad Data Leads to Good Models

2025-05-07

AI Summary

This work challenges the conventional assumption that “cleaner pretraining data always yields better models,” investigating the synergistic relationship between pretraining data toxicity and post-training controllability. Method: Leveraging the Olmo-1B architecture, we conduct controlled pretraining, representation geometry analysis, and inference-time intervention (ITI) to systematically examine how toxic data influences internal representations and downstream controllability. Contribution/Results: We discover that moderate inclusion of toxic data enhances linear separability of toxicity-related features in the representation space, significantly improving post-training toxicity suppression efficacy. This leads to the first proposal of a “pretraining–post-training co-design” paradigm, revealing the counterintuitive principle that “bad data” can enhance model controllability. Empirical results show that models pretrained on toxic data—when subjected to ITI—achieve substantially lower toxicity scores on Toxigen and Real Toxicity Prompts benchmarks than clean-data baselines, while preserving general capabilities, thereby achieving a superior trade-off between toxicity mitigation and capability retention.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Adversarial Learning & Robustness
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of"quality"from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model's output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

Problem

• Examining how toxic pre-training data affects model control
• Studying data composition's impact on representation space geometry
• Exploring trade-offs between toxicity reduction and model capabilities

Innovation

• Pre-training on toxic data improves control
• Toxic data simplifies toxicity representation
• Inference-time intervention enhances detoxification efficiency