Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients

2026-06-23

AI Summary

This work challenges the prevailing assumption that the apparent universality of power-law exponents in neural scaling laws reflects fundamental constraints, demonstrating instead that model performance is predominantly governed by the prefactor—whose dependence on data and architecture remains poorly understood. We provide the first systematic analysis showing that the observed exponent universality arises from common mechanisms such as Softmax nonlinearity, representation superposition, and Transformer layer composition. Through theoretical analysis and scaling law modeling, we establish that current large language models belong to a universality class with a fixed exponent, thereby shifting the optimization focus toward controlling the prefactor. This reframing offers a novel pathway to transcend existing scaling paradigms by explicitly engineering architectural and data-related factors that shape the coefficient rather than the exponent.

Abstract

Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute. This position paper argues that the exponents of these power laws are fixed by generic mechanisms: a one-third time scaling due to the strong nonlinearity of Softmax, an inverse width scaling due to representational superposition, and an inverse depth scaling due to ensemble averaging of Transformer layers. These mechanisms are robust to a wide range of data structures and architectural details, placing current large language models in a universality class with fixed exponents. The coefficients, however, are expected to be sensitive to data and architecture details, and directly determine practical quantities such as the optimal model shape and the compute-optimal frontier. We therefore argue that understanding the coefficients is the key to near-term performance improvements, and that a closer examination of the current universality class may reveal pathways to better universality classes.

Problem

• neural scaling laws
• coefficients
• universality class
• large language models
• compute-optimal frontier

Innovation

• neural scaling laws
• universality class
• scaling exponents
• scaling coefficients
• large language models