AI-Assisted Systematization for Evaluating GenAI Systems

2026-05-25

AI Summary

This study addresses the challenge of effectively measuring abstract target concepts—such as “reasoning” or “fairness”—in generative AI evaluation, which are often hindered by ambiguous definitions. To tackle this issue, the authors propose an AI-assisted, systematic methodology that introduces, for the first time, “concept specifications” and validation worksheets to translate nebulous constructs into structured, measurable evaluation frameworks. The approach employs two types of systematizers—zero-shot and multi-agent—that leverage large language models to emulate human modeling processes, substantially enhancing both the efficiency and quality of the translation. Empirical validation through case studies on “hate speech” and “digital empathy” demonstrates that the generated concept specifications exhibit strong content validity and information recoverability, confirming the method’s effectiveness and practical utility.

Abstract

Evaluating generative AI (GenAI) systems is challenging because many targets of evaluation are broad, contested concepts, such as "reasoning," "fairness," or "creativity." When these concepts are left underspecified, it becomes unclear what should be measured or how evaluation results should be interpreted. This problem reflects a missing step: systematization, that is, moving from a broad background concept to an explicit, structured account of the concept in measurable terms. To help address the fact that systematization is cognitively demanding and resource-intensive, we investigate whether AI assistance can support this process. To enable AI-assisted systematization and assess its quality, we introduce a structured representation of a systematized concept, a concept spec, and a validation worksheet. We then develop two AI-assisted systematizers: a direct, zero-shot approach and a multi-agent approach that more closely mirrors manual systematization approaches from existing literature. We use these systematizers to produce concept specs for two concepts -- hate-based rhetoric and digital empathy -- and evaluate resulting concept specs on content validity and information recoverability.

Problem

• generative AI
• evaluation
• systematization
• concept specification
• content validity

Innovation

• AI-assisted systematization
• concept spec
• generative AI evaluation
• multi-agent approach
• content validity