Distributional Scaling Laws for Emergent Capabilities

2025-02-24

AI Summary

This paper investigates the nature of “emergent abilities” in large language models (LLMs), questioning whether they reflect genuine qualitative leaps or are artifacts of measurement and statistical thresholds. Method: The authors introduce a *distributional scaling law* framework, leveraging multi-seed training, synthetic length-generalization tasks, and population-level performance distribution analysis on MMLU. Contribution/Results: They demonstrate that apparent emergence arises from continuous evolution of performance distributions—particularly bimodal ones—with threshold-based binarization inducing illusory discontinuities. For the first time, inverse scaling is explained as the coupled effect of rising mean accuracy among successful samples and declining overall success probability. Experiments reproduce coexisting linear and emergent scaling behaviors on synthetic tasks and validate the framework on real LLM populations, offering a rigorous, distribution-centric perspective on LLM scaling behavior.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In this paper, we explore the nature of sudden breakthroughs in language model performance at scale, which stands in contrast to smooth improvements governed by scaling laws. While advocates of"emergence"view abrupt performance gains as capabilities unlocking at specific scales, others have suggested that they are produced by thresholding effects and alleviated by continuous metrics. We propose that breakthroughs are instead driven by continuous changes in the probability distribution of training outcomes, particularly when performance is bimodally distributed across random seeds. In synthetic length generalization tasks, we show that different random seeds can produce either highly linear or emergent scaling trends. We reveal that sharp breakthroughs in metrics are produced by underlying continuous changes in their distribution across seeds. Furthermore, we provide a case study of inverse scaling and show that even as the probability of a successful run declines, the average performance of a successful run continues to increase monotonically. We validate our distributional scaling framework on realistic settings by measuring MMLU performance in LLM populations. These insights emphasize the role of random variation in the effect of scale on LLM capabilities.

Problem

• Explores sudden performance breakthroughs in language models
• Investigates continuous probability distribution changes in training
• Validates distributional scaling framework on LLM performance

Innovation

• Continuous distribution changes drive breakthroughs
• Random seeds influence scaling trends
• MMLU validates distributional scaling framework