AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

2026-05-12

AI Summary

This work addresses a critical limitation in current evaluations of software engineering agents, which rely solely on binary pass/fail outcomes and thereby risk misclassifying low-quality yet fortuitously successful executions—termed “lucky passes”—as valid solutions. To mitigate this, the authors introduce AgentLens, a novel framework that formally defines and exposes the “lucky pass” phenomenon through a three-tiered process-quality assessment. Leveraging 2,614 agent execution trajectories, they construct AgentLens-Bench, a benchmark dataset integrating a prefix-tree acceptor (PTA) and a context-aware intent classifier to establish task-level behavioral reference models and intent annotations. Analysis of 1,815 annotated trajectories reveals that 10.7% constitute lucky passes, with model-specific lucky-pass rates ranging from 0.5% to 23.2%. Re-ranking models by process quality shifts some rankings by as many as five positions, underscoring the impact of evaluation beyond mere correctness.

Abstract

Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view treats a principled solution and a chaotic trial-and-error process as equivalent. We show that this equivalence is empirically false. We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks. Of these, 47 have enough passing trajectories to construct task-level process references, yielding a 1,815-trajectory evaluation subset. Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass: regression cycles, blind retries, missing verification, or temporally disordered exploration, implementation, and verification. We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories, and release AgentLens-Bench, a dataset of 1,815 trajectories annotated with quality scores, waste signals, divergence points, and 47 task-level Prefix Tree Acceptor (PTA) references. AgentLens builds PTA references by merging multiple passing solutions for the same task, and uses a context-sensitive intent labeler to assign actions to Exploration, Implementation, Verification, or Orchestration based on trajectory history rather than tool identity alone. On AgentLens-Bench, the quality score separates passing trajectories into Lucky, Solid, and Ideal tiers and further decomposes Lucky Passes into five recurring mechanisms. Across the eight model backends, Lucky rates range from 0.5% to 23.2%, and some models move by as many as five rank positions when ranked by quality score instead of pass rate. We release the anonymized project repository, including the AgentLens-Bench dataset and AgentLens SDK, at https://github.com/microsoft/code-agent-state-trajectories/.

Problem

• SWE-Agent Evaluation
• Lucky Pass
• Process-level Assessment
• Trajectory Analysis
• Software Engineering Agents

Innovation

• AgentLens
• process-level evaluation
• Lucky Pass
• Prefix Tree Acceptor
• SWE-agent trajectories