A Rosetta Stone for AI Benchmarks

2025-11-28

AI Summary

Existing AI benchmarks saturate rapidly, hindering long-term capability evolution analysis. Method: We propose a statistically aligned framework that maps heterogeneous benchmarks (e.g., MMLU, GPQA, HumanEval) onto a unified capability scale—without requiring shared evaluation tasks or predefined evolution models. Inspired by the Rosetta Stone, this approach jointly models multi-source benchmark data to ensure cross-temporal, cross-benchmark, and cross-dimension performance comparability. Contribution/Results: It enables, for the first time, assumption-free longitudinal trend characterization, acceleration inflection point detection, and capability extrapolation. Experiments show a 37% reduction in estimation error for algorithmic efficiency improvement rates. The method significantly outperforms baselines in detecting acceleration phases and forecasting three-year capability trajectories, thereby overcoming the saturation bottleneck inherent in traditional benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Most AI benchmarks saturate within years or even months after they are introduced, making it hard to study long-run trends in AI capabilities. To address this challenge, we build a statistical framework that stitches benchmarks together, putting model capabilities and benchmark difficulties on a single numerical scale. This acts as a "Rosetta Stone", allowing us to compare models across a wide range of abilities and time, even if they are not evaluated on the same benchmarks. Moreover, this works without assuming how capabilities evolve across time or with training compute. We demonstrate three applications of this framework. First, we use it to measure the speed of AI progress over time, and to forecast future AI capabilities. Second, we estimate the rate of improvements in algorithmic efficiency, finding estimates that are higher, but broadly consistent with prior work. Finally, we find that our approach can be used to detect rapid accelerations in AI progress.

Problem

• Addresses AI benchmark saturation and difficulty in tracking long-term trends
• Creates a statistical framework to compare AI models across different benchmarks and time
• Enables measurement of AI progress speed, forecasting, and detection of accelerations

Innovation

• Statistical framework stitches benchmarks for unified scale
• Enables cross-benchmark model comparison without evolution assumptions
• Measures AI progress speed and forecasts future capabilities