Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead

2025-07-30

AI Summary

Directly applying human-centric intelligence and personality assessments to evaluate large language models (LLMs) constitutes an ontological misapplication, resulting in poor validity, cultural bias, data contamination, and prompt sensitivity. Method: The study advocates abandoning anthropomorphic evaluation paradigms in favor of a theoretically grounded, empirically validated, AI-specific assessment framework. Drawing on psychometric test development principles, it pursues dual strategies—adapting existing instruments and designing novel, AI-native paradigms—while rigorously aligning metric definitions with AI behavioral logic rather than human cognitive architecture. Contribution/Results: The work establishes “system alignment” as the foundational evaluation philosophy, supplanting “human benchmarking.” This provides a principled methodological foundation for AI capability assessment, advancing the field from ad hoc, empirically driven testing toward a theory-informed, scientific measurement science.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Philosophy and Ethics of AI: Morality & Value-based AI
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models (LLMs) have achieved remarkable results on a range of standardized tests originally designed to assess human cognitive and psychological traits, such as intelligence and personality. While these results are often interpreted as strong evidence of human-like characteristics in LLMs, this paper argues that such interpretations constitute an ontological error. Human psychological and educational tests are theory-driven measurement instruments, calibrated to a specific human population. Applying these tests to non-human subjects without empirical validation, risks mischaracterizing what is being measured. Furthermore, a growing trend frames AI performance on benchmarks as measurements of traits such as ``intelligence'', despite known issues with validity, data contamination, cultural bias and sensitivity to superficial prompt changes. We argue that interpreting benchmark performance as measurements of human-like traits, lacks sufficient theoretical and empirical justification. This leads to our position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead. We call for the development of principled, AI-specific evaluation frameworks tailored to AI systems. Such frameworks might build on existing frameworks for constructing and validating psychometrics tests, or could be created entirely from scratch to fit the unique context of AI.

Problem

• Human tests mischaracterize AI capabilities due to ontological mismatch
• AI benchmark performance lacks validity as human trait measurement
• Need AI-specific evaluation frameworks for accurate assessment

Innovation

• Develop AI-specific evaluation frameworks
• Avoid human psychological test misinterpretation
• Create principled tests for AI traits