AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation

2026-04-20

AI Summary

Existing rule-based or LLM-as-a-Judge approaches struggle to reliably verify agent behavior in complex environments and exhibit limited generalization. This work proposes AJ-Bench, the first Agent-as-a-Judge evaluation benchmark designed for environment-aware assessment, wherein judging agents actively interact with the environment and tools to gather verifiable evidence. The benchmark systematically evaluates agents’ capabilities in information acquisition and process verification across 155 tasks and 516 annotated trajectories spanning search, data systems, and graphical user interfaces. Experimental results demonstrate that this approach significantly outperforms conventional baselines and uncovers key challenges and open problems in agent-based verification.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Mechanism Design
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

As reinforcement learning continues to scale the training of large language model-based agents, reliably verifying agent behaviors in complex environments has become increasingly challenging. Existing approaches rely on rule-based verifiers or LLM-as-a-Judge models, which struggle to generalize beyond narrow domains. Agent-as-a-Judge addresses this limitation by actively interacting with environments and tools to acquire verifiable evidence, yet its capabilities remain underexplored. We introduce a benchmark AJ-Bench to systematically evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories. The benchmark comprehensively assesses judge agents' abilities in information acquisition, state verification, and process verification. Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification. Our data and code are available at https://aj-bench.github.io/.

Problem

• Agent-as-a-Judge
• environment-aware evaluation
• reinforcement learning
• behavior verification
• generalization

Innovation

• Agent-as-a-Judge
• environment-aware evaluation
• reinforcement learning
• benchmarking
• verifiable evidence