Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation

2026-05-08

AI Summary

This study addresses the ill-posedness arising from fitting large language model scaling laws under a fixed tokens-per-parameter (TPP) ratio, which induces severe collinearity in the design matrix and consequently leads to parameter non-identifiability and unreliable extrapolation. For the first time, the work reveals this pathological behavior through the geometric lens of the Jacobian and leverages condition number theory to derive a TPP diversity threshold that guarantees well-posed identifiability. Validated via high-dimensional nonlinear least squares analysis and extensive experiments across multiple corpora and numerical precisions, the proposed non-collinear training design significantly outperforms conventional approaches—achieving a 97.3% win rate across four scaling laws and five corpora—while substantially narrowing confidence intervals and enhancing extrapolation accuracy.

Abstract

Neural scaling laws approximate a language model's loss as a power-law function of parameter count $N$ and token count $D$. Following Chinchilla-style compute-optimal training, many studies fit scaling laws from runs performed under a fixed tokens-per-parameter (TPP) ratio $k$ and set $D = kN$. We show that this collinear design, combined with the empirically common near-equality of the exponents governing $N$ and $D$, induces an inherent ill-conditioning in the Gauss-Newton least-squares problem: the condition number of the design grows as the inverse square of the gap between the $N$ and $D$-exponents. The scale coefficients become practically unidentifiable, with confidence intervals inflating by an order of magnitude or more, yielding a ``sloppy'' model whose extrapolations degrade sharply off the training ray. We prove this for four scaling-law formalisms and derive a closed-form TPP-diversity threshold that is necessary and sufficient for well-conditioned estimation. Empirically, non-collinear designs outperform collinear ones on held-out splits with a 97.3\% win rate across four laws, five corpora, multiple floating point precision modes. We further show the degeneracy is rooted in Jacobian geometry and is not an artifact of the loss function: any smooth estimation objective whose curvature involves the Jacobian inherits the same ill-conditioning.

Problem

• scaling laws
• tokens-per-parameter
• ill-conditioning
• extrapolation
• language models

Innovation

• scaling laws
• tokens-per-parameter
• ill-conditioning
• extrapolation robustness
• non-collinear design