Domain-Aware Scaling Laws Uncover Data Synergy

2026-07-12

AI Summary

This work addresses a critical limitation of conventional scaling laws, which overlook the influence of data domain composition on model performance and thus fail to account for nonlinear synergistic or interference effects arising from cross-domain data mixing. The study formally defines and empirically quantifies both direct and second-order inter-domain interactions during language model pretraining. It introduces a domain-aware scaling framework that leverages regression modeling and ablation studies to analyze multi-domain mixing strategies. This framework substantially improves the accuracy of performance prediction and demonstrates strong empirical validity by correctly ranking the effectiveness of different data mixing configurations in actual training runs, thereby overcoming the constraints of domain-agnostic scaling laws.

Abstract

Machine learning progress is often attributed to scaling model size and dataset volume, yet the composition of data can be just as consequential. Empirical findings repeatedly show that combining datasets from different domains yields nontrivial interactions. For instance, adding code improves mathematical reasoning, while certain mixtures introduce interference that reduces model performance. We refer to these effects collectively as data synergy, where the contribution of multiple domains exceeds or falls short of the sum of their isolated contributions. In this work, we formalize and quantify data synergy in language model pretraining. Leveraging observational variation across open-weight LLMs with diverse pretraining mixtures, we estimate both direct domain-to-benchmark synergy (how one domain contributes to performance on another) and a second-order domain-domain synergy (capabilities that require co-occurrence of multiple domains). Our framework improves predictive accuracy over domain-agnostic scaling laws and recovers stable synergy estimates. We validate these estimates by training models on predicted optimal and predicted anti-optimal mixtures and confirm that our synergy estimates correctly predict performance rankings.

Problem

• data synergy
• domain composition
• scaling laws
• language model pretraining
• dataset mixture

Innovation

• data synergy
• domain-aware scaling laws
• language model pretraining
• domain-domain interaction
• scaling laws