Two AI Metrics Diverged: Will it Make All the Difference?

2026-07-01

AI Summary

This study investigates whether the performance gap between large-scale and small, low-cost models will continue to widen or eventually converge amid sustained growth in computational resources. By constructing a classification framework grounded in the functional forms of performance metrics and integrating mathematical modeling, compute scaling laws, and multidimensional capability evaluation, the work systematically analyzes the relationship between training/inference compute and various performance indicators. The findings reveal that bounded metrics inherently favor the widespread adoption of small models, whereas unbounded metrics—particularly those tied to critical capabilities such as software engineering—concentrate high performance among a few resource-rich entities. This research underscores the pivotal role of metric choice in shaping AI development trajectories and policy decisions, while rigorously delineating the conditions under which small models can remain competitive.

Abstract

As exponential compute scaling continues, will the capabilities of frontier AI models outstrip what is accessible to developers on a small fixed budget? Or will capabilities converge, with "meek models inheriting the earth"? Building on Gundlach et al. (2025b), we show that the answer depends on how we value and measure AI capabilities. We discuss conventional performance measures and show that, while validation loss shows a shrinking gap, on other metrics frontier models grow their lead forever. Classifying performance metrics by their functional forms in relation to training (and inference) compute, we provide tight mathematical conditions for determining which metrics favor meek models, and show that bounded performance metrics always do. But careful interpretation of performance metrics is essential: we show that many common bounded metrics have closely-related counterpart metrics that are unbounded (and vice versa). Determining the apt metric in a domain is a prerequisite for policy, since bounded and unbounded metrics may suggest opposing policy responses. If a particular capability -- like software engineering, synthetic biology, or rhetorical persuasiveness -- is unbounded when measured in the terms we care about, frontier-level capability will likely be concentrated in the hands of a few wealthy actors. Conversely, if that capability is instead bounded, frontier-level capabilities proliferate through meek models into the hands of the many.

Problem

• AI capabilities
• performance metrics
• compute scaling
• bounded metrics
• frontier models

Innovation

• bounded metrics
• unbounded metrics
• compute scaling
• performance evaluation
• meek models