Amortizing Scaling Law Construction Costs

2026-09-04

AI Summary

本文提出一种高效构建缩放定律的框架，通过贝叶斯优化问题和计算预算逐步扩展的方法，大幅降低计算成本并准确拟合缩放定律。

Abstract

Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to $10\text{--}100\times$.

Problem

• Scaling Laws
• Large Foundation Models
• Computational Cost
• Hyperparameters
• Token Budgets

Innovation

• Bayesian optimization
• scaling law construction
• compute budget expansion
• surrogate model