How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents

2026-09-16

AI Summary

研究通过模型增长、递归和边界操作符调整预训练中的缩放指数，提高计算效率，使性能随计算量增加而显著提升。

Abstract

Scaling laws predict how loss decreases with increases in computation. We show, contrary to conventional wisdom, that architectural interventions can modify scaling exponents in pre-training, leading to exponential improvements in performance with increases in computation. As an anchoring point, we consider the architectural formulation of looped transformers. Although not typically used in this way, looping, also known as recursive depth, provides a mechanism for model growth, by increasing the number of loops during training. Model growth, with and without shared weights, provides the biggest changes to the scaling exponents. In particular, a 7.4B model growth architecture matches GPT-3 13B on CORE with roughly $20\times$ less compute, and has compute efficiency gains that increase with scale. Moreover, simply using a boundary operator in a vanilla transformer, which normalizes and injects an earlier block, also provides increasing compute-efficiency gains, although to a lesser extent. In the data-constrained, multi-epoch setting, standard looping has a useful regularizing effect, where we find it is compute-optimal to increase the number of loops with scale. These results can be understood through the lens of computational depth: for a given computational budget, we wish to increase the usable depth of the transformer, which can lead to efficiency gains that increase with scale.

Problem

• Scaling Exponents
• Model Growth
• Recursion
• Boundary Operators
• Computational Depth

Innovation

• model growth
• scaling exponents
• recursive depth
• boundary operator
• computational efficiency