AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

2026-07-07

AI Summary

Existing evaluations of code agents predominantly focus on task success or failure, overlooking the nuanced interaction trajectories observed in real-world usage. This work proposes the first fine-grained evaluation framework that integrates formal verification with large language model–generated commentary, leveraging complete interaction traces from actual production environments to establish a multidimensional and interpretable assessment mechanism. The framework enables detailed model behavior diagnostics, supports regression detection across model versions, and incorporates an automated evaluation pipeline. Its effectiveness has been validated through internal agent iterations and nightly regression testing, and the associated benchmark suite has been open-sourced to provide the research community with a high-value evaluation tool.

Abstract

We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at https://github.com/agent-lens/agent-lens-bench.

Problem

• code agent evaluation
• trajectory assessment
• interactive coding
• benchmarking
• agent behavior

Innovation

• trajectory evaluation
• interactive coding agents
• production-assessed benchmark
• LLM-based review
• agent behavior diagnosis