From Human-Level AI Tales to AI Leveling Human Scales

2026-02-21

AI Summary

Current AI models are often inaccurately described as achieving “human-level” performance due to the absence of evaluation benchmarks grounded in the global distribution of human capabilities. This work proposes a human-anchored evaluation framework that leverages large-scale international assessments—such as PISA and TIMSS—and integrates stratified sampling with post-stratification techniques to extrapolate ability distributions across diverse populations. The framework constructs a multidimensional logit scale, enabling, for the first time, direct comparison of AI performance against the full spectrum of human abilities on a unified, recalibratable metric. Consequently, AI performance can be objectively expressed as the log-odds corresponding to the probability of success among the global human population, establishing a standardized, cross-task, and cross-population evaluation benchmark.

Technology Category

• Humans and AI: AI for Accessibility
• Machine Learning: Evaluation and Analysis
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Comparing AI models to "human level" is often misleading when benchmark scores are incommensurate or human baselines are drawn from a narrow population. To address this, we propose a framework that calibrates items against the 'world population' and report performance on a common, human-anchored scale. Concretely, we build on a set of multi-level scales for different capabilities where each level should represent a probability of success of the whole world population on a logarithmic scale with a base $B$. We calibrate each scale for each capability (reasoning, comprehension, knowledge, volume, etc.) by compiling publicly released human test data spanning education and reasoning benchmarks (PISA, TIMSS, ICAR, UKBioBank, and ReliabilityBench). The base $B$ is estimated by extrapolating between samples with two demographic profiles using LLMs, with the hypothesis that they condense rich information about human populations. We evaluate the quality of different mappings using group slicing and post-stratification. The new techniques allow for the recalibration and standardization of scales relative to the whole-world population.

Problem

• human-level AI
• benchmarking
• population calibration
• scale standardization
• AI evaluation

Innovation

• human-anchored scale
• population calibration
• logarithmic proficiency scale
• large language models
• post-stratification