Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents

2026-05-21

AI Summary

Existing evaluation methods struggle to effectively supervise increasingly autonomous LLM agents, often constrained by static error taxonomies or insufficient assessment capabilities. This work proposes Agentic CLEAR, a novel framework that introduces the first dynamic, multi-granular, data-driven, and domain-adaptive evaluation mechanism for agentic systems. It automatically models agent behavior across three levels—system, trajectory, and node—and generates interpretable textual feedback. Integrated with an intuitive user interface, the framework ensures high usability and seamless deployment. Extensive experiments across four benchmarks, seven agent types, and tens of thousands of LLM calls demonstrate that the framework’s feedback aligns closely with human annotations and accurately predicts task success rates.

Abstract

Agentic systems are becoming more capable: agents define strategies, take actions, and interact with different environments. This autonomy poses serious challenges for overseeing and assessing agent behavior. Most current tools are limited, focusing on observability with basic evaluation capabilities or imposing static, hand-crafted error taxonomies that cannot adapt to new domains. To address this gap, we present Agentic CLEAR, an automatic, dynamic, and easy-to-use evaluation framework. It produces textual insights into the agent behavior on three levels of granularity: system, trace, and node. Agentic CLEAR operates above the observability layer, enabling seamless integration and featuring an intuitive UI that makes agent evaluation highly accessible. In our experiments on four benchmarks, seven agentic settings, and tens of thousands of LLM calls, we show that Agentic CLEAR produces high-quality, data-driven, insightful feedback. Our analysis shows strong alignment with human-annotated errors and the ability to predict task success rate.

Problem

• LLM agents
• evaluation framework
• multi-level evaluation
• agent behavior
• error taxonomy

Innovation

• Agentic evaluation
• multi-level analysis
• dynamic error taxonomy
• LLM agents
• automated feedback