Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

2026-07-10

AI Summary

This study addresses a critical gap in the evaluation of command-line interface (CLI) coding agents by reframing failure not as a static endpoint but as a temporally evolving phenomenon. Introducing a process-oriented analytical framework, the authors conduct a large-scale empirical analysis of 1,794 complete execution trajectories—comprising over 63,000 steps—across seven state-of-the-art large language models and three agent frameworks. Through meticulous human annotation of failure onset, root causes, recovery attempts, and system consistency, the work uncovers fourteen key findings. These reveal that failures predominantly originate from early cognitive errors, often manifesting within the first few steps and rapidly cascading into irrecoverable states. The results underscore the inadequacy of outcome-only evaluation paradigms and highlight the pivotal role of early intervention in enhancing agent reliability.

Abstract

Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a growing concern. Existing empirical studies investigate why coding agents fail, yet they largely treat failure as a final outcome rather than a temporal process, providing limited insight into how failures emerge, evolve, and become unrecoverable. We present the first large-scale empirical study of CLI coding-agent failure trajectories, introducing a process-oriented framework that analyzes failure through its onset, evolution, and recovery across execution trajectories. We first collect 3,843 execution trajectories generated by seven frontier models across three coding-agent scaffolds (OpenHands, MiniSWE, and Terminus2) on Terminal-Bench, then carefully filter them to obtain 1,794 complete and valid trajectories for manual annotation (over 63,000 execution steps), from which we derive 14 findings spanning failure occurrence, root causes, recovery, and cross-system consistency. Our findings show that coding-agent failures are predominantly driven by epistemic errors, typically begin within the first few execution steps, and often remain hidden until recovery is no longer possible, suggesting that improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.

Problem

• failure trajectories
• coding agents
• large language models
• empirical study
• CLI environments

Innovation

• failure trajectory
• process-oriented analysis
• CLI coding agent
• epistemic error
• execution trace annotation