From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis

2025-12-15

AI Summary

Empirical neural scaling laws lack a rigorous theoretical foundation rooted in linguistic principles. Method: We establish a formal deductive chain linking Zipf’s law—governing word-frequency distributions—to neural scaling laws, mediated by Heaps’ law (vocabulary growth) and Hilberg’s hypothesis (entropy scaling). Using statistical language modeling, information-theoretic analysis, and asymptotic derivation, we validate the framework via the Santa Fe process and analytically solvable toy models. Contribution/Results: We present the first complete, mathematically rigorous derivation showing that neural scaling laws—describing the asymptotic decay of cross-entropy rate with training data volume, parameter count, and compute—are necessary information-theoretic consequences of Zipf’s law under broad assumptions. This work identifies intrinsic linguistic statistics—not engineering heuristics—as the fundamental origin of scaling behavior, providing the first linguistically grounded theoretical explanation for large-model empirical regularities.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Information Theory
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

We inspect the deductive connection between the neural scaling law and Zipf's law -- two statements discussed in machine learning and quantitative linguistics. The neural scaling law describes how the cross entropy rate of a foundation model -- such as a large language model -- changes with respect to the amount of training tokens, parameters, and compute. By contrast, Zipf's law posits that the distribution of tokens exhibits a power law tail. Whereas similar claims have been made in more specific settings, we show that the neural scaling law is a consequence of Zipf's law under certain broad assumptions that we reveal systematically. The derivation steps are as follows: We derive Heaps' law on the vocabulary growth from Zipf's law, Hilberg's hypothesis on the entropy scaling from Heaps' law, and the neural scaling from Hilberg's hypothesis. We illustrate these inference steps by a toy example of the Santa Fe process that satisfies all the four statistical laws.

Problem

• Connects neural scaling law to Zipf's law via Heaps' law and Hilberg's hypothesis
• Derives neural scaling from Zipf's law under broad systematic assumptions
• Illustrates inference steps using a toy Santa Fe process example

Innovation

• Deriving neural scaling law from Zipf's law via Heaps' law
• Linking vocabulary growth to entropy scaling through Hilberg's hypothesis
• Validating inference steps using Santa Fe process as a toy example