Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents

2025-08-24

AI Summary

Current LLM agent evaluation relies on static benchmarks and small-scale manual assessments, lacking efficient, comprehensive, and automated testing methodologies. To address this, we propose an automated testing and evaluation meta-agent tailored for conversational AI agents. Our approach innovatively integrates role-driven adversarial test case generation, adaptive difficulty adjustment, multi-source evidence synthesis (including code analysis, design intent extraction, and literature retrieval), and dynamic LLM-as-a-Judge scoring, all within a closed-loop feedback framework. Evaluated on travel planning and Wikipedia writing agents, our method identifies defects with diversity and severity comparable to expert annotation, reduces testing time from days to 20–30 minutes, and concurrently produces interpretable quantitative metrics and qualitative diagnostic reports. This significantly enhances the precision and efficiency of capability gap identification while improving agent trustworthiness.

Technology Category

• Multiagent Systems: Adversarial Agents
• Search and Optimization: Metareasoning and Metaheuristics
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

LLM agents are increasingly deployed to plan, retrieve, and write with tools, yet evaluation still leans on static benchmarks and small human studies. We present the Agent-Testing Agent (ATA), a meta-agent that combines static code analysis, designer interrogation, literature mining, and persona-driven adversarial test generation whose difficulty adapts via judge feedback. Each dialogue is scored with an LLM-as-a-Judge (LAAJ) rubric and used to steer subsequent tests toward the agent's weakest capabilities. On a travel planner and a Wikipedia writer, the ATA surfaces more diverse and severe failures than expert annotators while matching severity, and finishes in 20--30 minutes versus ten-annotator rounds that took days. Ablating code analysis and web search increases variance and miscalibration, underscoring the value of evidence-grounded test generation. The ATA outputs quantitative metrics and qualitative bug reports for developers. We release the full methodology and open-source implementation for reproducible agent testing: https://github.com/KhalilMrini/Agent-Testing-Agent

Problem

• Automated testing for conversational AI agents' capabilities
• Dynamic evaluation replacing static benchmarks and human studies
• Generating adaptive difficulty tests to identify agent weaknesses

Innovation

• Meta-agent combines static analysis and adversarial generation
• Adaptive test difficulty via LLM-as-a-Judge feedback
• Generates evidence-grounded tests with quantitative metrics