Toward an Evaluation Science for Generative AI Systems

2025-03-07

AI Summary

Current generative AI systems suffer from inadequate evaluation: static benchmarks fail to reflect real-world performance, and case-by-case audits lack scalability. To address this, we propose a scientific framework for evaluating generative AI—shifting assessment from fragmented auditing toward systematic, iterative, and institutionalized engineering practice. Our method introduces three foundational principles: (1) evaluation metrics must empirically capture real-world performance; (2) metrics must evolve dynamically over time; and (3) assessment must be conducted by specialized, accredited institutions operating under rigorous procedural standards. Drawing cross-domain insights from transportation, aerospace, and pharmaceutical safety regulation, we integrate systems safety engineering, metrology, and institutional design. The framework provides both theoretical grounding and actionable pathways for AI safety verification, regulatory alignment, and responsible industrial deployment.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Safety and Robustness
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

There is an increasing imperative to anticipate and understand the performance and safety of generative AI systems in real-world deployment contexts. However, the current evaluation ecosystem is insufficient: Commonly used static benchmarks face validity challenges, and ad hoc case-by-case audits rarely scale. In this piece, we advocate for maturing an evaluation science for generative AI systems. While generative AI creates unique challenges for system safety engineering and measurement science, the field can draw valuable insights from the development of safety evaluation practices in other fields, including transportation, aerospace, and pharmaceutical engineering. In particular, we present three key lessons: Evaluation metrics must be applicable to real-world performance, metrics must be iteratively refined, and evaluation institutions and norms must be established. Applying these insights, we outline a concrete path toward a more rigorous approach for evaluating generative AI systems.

Problem

• Develop evaluation metrics for real-world generative AI performance
• Iteratively refine metrics to improve generative AI safety
• Establish institutions and norms for generative AI evaluation

Innovation

• Real-world applicable evaluation metrics
• Iterative refinement of metrics
• Establishment of evaluation institutions