Unified Neural Scaling Laws

2026-05-25

AI Summary

Existing neural scaling laws struggle to accurately capture the complex interplay among model performance and multiple co-varying factors such as parameter count, dataset size, training and inference steps, compute budget, and hyperparameters. This work proposes the Unified Neural Scaling Law (UNSL), a general functional form that jointly models multidimensional scaling behaviors across diverse architectures and tasks—including vision, language, mathematics, and reinforcement learning. Grounded in extensive empirical data spanning multiple tasks and architectures, UNSL integrates both upstream and downstream performance metrics, enabling, for the first time, highly accurate fitting and extrapolation of model performance under simultaneous multidimensional scaling. Experimental results demonstrate that UNSL substantially outperforms existing scaling laws in large-scale multitask settings, significantly improving extrapolation accuracy.

Abstract

We present a functional form (that we refer to as a Unified Neural Scaling Law (UNSL)) that accurately models and extrapolates the scaling behaviors of deep neural networks as multiple dimensions all vary simultaneously (i.e. how the evaluation metric of interest varies as one simultaneously varies the number of model parameters, training dataset size, number of training steps, number of inference steps, amount of compute, and various hyperparameters) for various architectures and for each of various tasks within a varied set of upstream and downstream tasks. This set includes large-scale vision, language, math, and reinforcement learning. When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set.

Problem

• neural scaling
• extrapolation
• multi-dimensional scaling
• deep neural networks
• scaling laws

Innovation

• Unified Neural Scaling Law
• neural scaling
• extrapolation
• multi-dimensional scaling
• deep learning