Validated Hypotheses as a Lens for Human-Likeness Evaluation in AI Agents

2026-05-14

AI Summary

This study addresses the absence of an objective, decomposable, and scalable framework for evaluating the behavioral alignment of large language model agents with human behavior. The authors propose a novel evaluation paradigm grounded in well-established, replicable behavioral hypotheses from social science, systematically translating human-subject experiments into a standardized benchmark for AI agents. They introduce HumanStudy-Bench, an open platform, along with two new metrics—Probability Alignment Score (PAS) and Effect Consistency Score (ECS)—to quantify the consistency between agents and human populations in both inference patterns and effect sizes. Evaluating four agent designs across ten models in twelve robust experiments reveals a bimodal performance distribution, with agent architecture exerting a stronger—and non-monotonic—influence on human alignment than model scale.

Abstract

We propose using validated behavioral hypotheses as a lens for evaluating human-likeness in LLM-based agents. Our key idea is simple: If an agent is human-like, a population of such agents should reach the same inferential conclusion as the human population when run through the same experiment. Decades of social science have produced many such validated findings, each anchored to concrete experimental protocols and robustly established through independent replication. This yields an evaluation that is objective, decomposable, and scalable. We operationalize this lens through HumanStudy-Bench, an open platform that turns published human-subject studies into reusable simulation environments and administers the evaluation to configurable agents. It scores agent-human alignment on two metrics: the Probability Alignment Score (PAS) for inferential agreement and the Effect Consistency Score (ECS) for effect-size agreement. We curated an initial suite of 12 studies whose hypotheses are robustly established through independent replication, and evaluated 10 models under 4 agent designs. Results show that agent responses polarize between full replication and complete failure; agent design influences alignment more than model scale, but its effect is non-monotonic.

Problem

• human-likeness evaluation
• AI agents
• behavioral hypotheses
• LLM-based agents
• agent-human alignment

Innovation

• human-likeness evaluation
• validated behavioral hypotheses
• HumanStudy-Bench
• Probability Alignment Score (PAS)
• Effect Consistency Score (ECS)