On the Smallness of the Large Language Models Scaling Exponents

2026-06-23

AI Summary

This study addresses the unsustainable energy consumption of current large language models (LLMs), attributed to suboptimal scaling exponents. It demonstrates that even under the assumption of infinitely growing training data, the loss function exhibits a non-zero asymptotic floor, fundamentally limiting the efficacy of conventional scaling laws in improving energy efficiency. Drawing an analogy from fluid turbulence phenomenology, the work reveals the critical role of data smoothness in shaping scaling behavior. Through theoretical analysis, asymptotic loss modeling, and cross-domain analogies, the paper argues that existing correction strategies are insufficient to resolve the sustainability challenges of LLM scaling and offers a novel perspective on how intrinsic data properties constrain the scalability of these models.

Abstract

We discuss reasons why the scaling exponents of current Large Language Models (LLMs) applications are indicating an unsustainable regime in terms of energy resources. We further show that attributing the smallness of such exponents to a numerical bias due to the neglect of a non-zero value of the loss function in the limit of infinite data (``pedestal effect") does not remove the unsustainability issue. Finally, the effects of the smoothness (roughness) of the data on the scaling exponents is commented upon based on an analogy with phenomenological models of fluid turbulence.

Problem

• scaling exponents
• Large Language Models
• unsustainability
• energy resources
• pedestal effect

Innovation

• scaling exponents
• large language models
• pedestal effect
• energy sustainability
• data roughness