One Spectrum, Two Resources: Data-Memory Scaling in Autoregressive Prediction

2026-09-11

AI Summary

研究探讨了自回归预测中数据与记忆资源的关系，通过预测能量谱方法证明了两者之间的最小最大定律，并通过实验验证了数据-记忆崩溃和耦合指数。

Abstract

How much learned memory is needed to benefit from more data? We show that the two resources are governed by one predictive-energy spectrum in a positive-entropy autoregressive retrieval source. Each coordinate contributes its query probability times the squared radius of its unknown logit. Writing $μ$ for the resulting energy spectrum, we prove the minimax law $\mathfrak R^*_{\rm value}(n,B)\asymp_R Φ_μ(n^{-1})+Φ_μ(τ_B), Φ_μ(t)=\int\min\{x,t\}\,μ(\mathrm dx),$ for $n$ prediction blocks and a learned state with at most $2^B$ values. Data set the resolution $1/n$; memory sets the level $τ_B$ reached by optimal bit allocation. The complete curve also recovers the positive spectrum. Energy-dimension pairing is essential: two causal sources with identical block-energy and block-dimension marginals have different data and memory exponents. A masked query-key attention head learns the route and values, realizing the law with explicit routing, format, and arithmetic errors. Further results give exponent-adaptive allocation, finite-precision realization, and compute-precision laws under two-sided arithmetic assumptions. Experiments recover the data-memory collapse and coupling exponents, explain the routing and allocation mechanisms, and examine weight-only quantization across six pretrained-model scales.

Problem

• autoregressive prediction
• data-memory scaling
• predictive-energy spectrum

Innovation

• predictive-energy spectrum
• minimax law
• masked query-key attention head
• exponent-adaptive allocation
• compute-precision laws