Measuring Intelligence Beyond Human Scale

2026-07-08

AI Summary

This work addresses the limitations of traditional human-centric benchmarks in evaluating artificial intelligence once systems surpass human performance. To overcome this, the paper proposes a novel evaluation paradigm based on relative measurement, wherein models autonomously generate public challenges and employ adversarial psychometrics to differentiate among competing systems. By integrating incentive mechanisms with an automated adjudication protocol, the framework enables scalable, human-judgment-free assessment. It is designed to support both verifiable tasks and open-domain scenarios, establishing—for the first time—a dynamic evaluation system that co-evolves with the capabilities of intelligent agents, thereby effectively measuring superhuman-level intelligence.

Abstract

How can we measure intelligence beyond human capability? Human-authored benchmarks saturate, and above human capability, examiners may not know which tasks are both hard and verifiable. We argue that this difficulty is inherent to absolute-scale evaluation and propose a new paradigm based on relative measurement in which models generate public challenges that separate other systems. Aggregating these outcomes yields an adversarial psychometric rating system that can scale with the systems being measured. We describe practical protocols that reduce incentives for private-information attacks, support judge-free adjudication, and naturally scale with agent capabilities. We instantiate the framework across verifiable and open-ended, non-verifiable domains, illustrating how model-generated evaluation can continue to measure systems beyond the human frontier.

Problem

• intelligence measurement
• beyond human scale
• benchmark saturation
• verifiable tasks
• psychometric evaluation

Innovation

• relative measurement
• adversarial psychometrics
• model-generated evaluation
• beyond-human intelligence
• judge-free adjudication