Towards Engineering Scaling Laws with Pretraining Data Composition

2026-06-18

AI Summary

This work addresses the strong dependence of traditional models on parameter scale in high-energy physics by introducing, for the first time in particle physics, an active control strategy over neural scaling laws for hadronic jet classification. By leveraging a high-fidelity simulator to generate low-cost synthetic data, the authors reconstruct the pretraining data distribution to better align with the downstream task while enhancing its diversity. This approach shifts the performance scaling paradigm from reliance on model size to dependence on data volume, yielding significantly improved classification accuracy under identical computational budgets. The study thus demonstrates effective engineering control over scaling behavior through deliberate manipulation of pretraining data composition.

Abstract

Neural scaling laws describe how model performance improves as a power law in compute, model size, and dataset size. While well-established for large language models, these relationships are emerging for large models in particle physics. As with language, empirical studies show that the performance scales as a power law. However, unlike natural language or image domains, fundamental physics has high-fidelity simulators that produce synthetic data cheaply. This favors scaling regimes where additional data is cheaper than additional parameters, and allows the pretraining dataset itself to be engineered to influence the scaling. For the task of classifying hadronic jets produced in collisions of high-energy particle beams, we show that the scaling behavior can be engineered towards requiring more data rather than larger models by inclusion of pretraining data which is more diverse and better aligned with the downstream classification task.

Problem

• neural scaling laws
• pretraining data composition
• hadronic jet classification
• data diversity
• model scaling

Innovation

• scaling laws
• pretraining data composition
• synthetic data
• jet classification
• data-efficient scaling