Counsel: A Meta-Evaluation Dataset for Agentic Tasks

2026-06-19

AI Summary

This work addresses the lack of reliable validation for existing automatic evaluation methods of agent task trajectories—such as LLM-as-a-judge—against human judgment. To bridge this gap, the authors introduce Counsel, the first meta-evaluation dataset specifically designed for agent tasks, which leverages open-source large language models to generate process-level critiques and employs crowdsourced annotations to produce three fine-grained types of human meta-evaluation labels. The dataset achieves a Krippendorff’s alpha inter-annotator agreement of 0.78 on the tau-bench and DA-Code benchmarks, enabling systematic calibration and alignment studies of automatic evaluators along two key dimensions: error localization accuracy and reasoning quality. Experimental results demonstrate that the best-performing evaluation model attains human alignment rates of approximately 88% and 65% on these respective metrics.

Abstract

As agentic systems tackle increasingly complex multi-step tasks, evaluating their trajectories presents a major bottleneck - human annotation of a single trajectory on popular agentic benchmarks can take hours, making it difficult to scale evaluations for measuring performance or curating training data. This has driven widespread reliance on automated approaches such as LLM-as-a-judge (LLMJ) to critique agents at the process and outcome-levels at scale, however, the soundness of LLMJ critiques often goes unmeasured. Here, we introduce Counsel, the first public dataset of meta-evaluations for agentic tasks. Counsel consists of process-level critiques from open-weight LLMJs on two agent benchmarks: tau-bench (customer support agents) and DA-Code (coding agents), and human meta-evaluations of these critiques. Human annotators label critiques on each flagged error as "spot on", "correct location but poor reasoning", or "should not have flagged", achieving reliable inter-annotator agreement (Krippendorff's alpha of 0.78). The resulting dataset stratifies LLMJ critiques by human alignment across both error location within a trajectory and reasoning quality, serving as valuable data to calibrate, improve, or train LLMJs for agents. Comparing open-weight judges, we find that more capable judge models and more reasoning effort both enabled improved human agreement, with the strongest judge reaching ~88% agreement on location and ~65% on reasoning. Counsel is generated using open-weight models and is permissively licensed for broad community use, which we hope will enable rigorous study and improved alignment of LLM-based evaluators for agentic systems.

Problem

• agentic evaluation
• LLM-as-a-judge
• meta-evaluation
• human alignment
• trajectory critique

Innovation

• meta-evaluation
• LLM-as-a-judge
• agentic systems
• process-level critique
• human alignment