Principles and Guidelines for Randomized Controlled Trials in AI Evaluation

2026-05-03

AI Summary

This study addresses the lack of standardized randomized controlled trial (RCT) frameworks in artificial intelligence evaluation, which has led to inconsistent research designs and limited reproducibility and comparability of results. Integrating Shadish’s four validity framework with TOP transparency guidelines, and drawing on RCT methodologies from clinical medicine, economics, psychology, and software engineering, this work proposes a structured RCT framework for AI that uniquely places human performance at the core of evaluation. It incorporates causal inference, heterogeneity analysis, and assessments of practical significance, while explicitly addressing AI-specific challenges such as model versioning, human–AI interaction, contamination effects, and fairness. The framework articulates five guiding principles and 33 actionable recommendations, supported by a tiered transparency mechanism that substantially enhances the rigor, reproducibility, and cross-study comparability of AI evaluation research, thereby establishing a foundational methodology for the field.

Abstract

This work establishes a foundational framework for standardizing AI evaluation RCTs (sometimes called human uplift studies). Drawing on established experimental practices from disciplines with established RCT traditions, including software engineering, economics, clinical and health sciences, and psychology, we adopt the (Shadish et al., 2002) four-validity framework and extend it with a fifth principle on transparency, repeatability, and verification adapted from the Transparency and Openness Promotion (TOP) Guidelines (Center for Open Science, 2025). We operationalize all five principles into 33 guidelines adapted for AI evaluation RCT contexts, expressed as requirements with rationales, implementation instructions, and evidence bases. We position the principles and guidelines as serving three key roles for AI evaluation RCTs: a design tool for planning studies, an evaluation rubric for assessing existing work, and a blueprint for standard setting as the field converges on norms. Our framework extends prior work by centering evaluation on human performance rather than model output alone, formalizing causal inference through RCT methodology for AI contexts, integrating heterogeneity analysis and practical significance assessment, implementing a graded transparency and repeatability framework, and addressing AI-specific challenges including model versioning, human-AI interaction dynamics, contamination and spillover effects, and equitable impact assessment.

Problem

• AI evaluation
• randomized controlled trials
• human performance
• causal inference
• transparency

Innovation

• randomized controlled trials
• AI evaluation
• human-centered assessment
• causal inference
• transparency and reproducibility