How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study

2026-07-12

AI Summary

This study addresses the limited understanding of how practitioners actually develop software engineering (SE) agents, particularly the lack of systematic investigation into the evolution of development workflows and core challenges. Through semi-structured interviews with 20 practitioners complemented by a survey of 80 respondents, this work proposes the first seven-stage workflow for SE agent development, revealing a paradigm shift toward “evaluation-driven iteration.” The research identifies that bottlenecks have moved beyond coding to non-coding tasks such as requirement specification, cross-role coordination, review, and deployment. It systematically characterizes six key challenges—including unreliable evaluation signals, accumulating comprehension debt, and behavioral drift induced by model updates—and synthesizes corresponding practical mitigation strategies.

Abstract

The rise of Software Engineering (SE) agents, i.e., LLM-based agents that can understand large codebases and carry out engineering tasks with limited human intervention, has been marked by rapid advances and adoption, but little is known about how developers build these systems in practice: existing studies mine repositories or examine deployment, but few investigate how SE agents are constructed. Through semi-structured interviews with 20 practitioners from 12 organizations and an online survey of 80 practitioners, this paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face. We find that as implementation becomes cheaper, bottlenecks shift rather than disappear: long-standing non-coding work such as requirements, coordination, review, and deployment becomes more visible, while reviewing and evaluating agent output becomes new and central. We characterize a seven-stage workflow and a shift toward evaluation-driven development, in which evaluation steers iteration and specifications become versioned artifacts read by both humans and agents. We further identify six challenges that teams face, together with the practices they adopt to address them, including unreliable evaluation signals, comprehension debt as code outpaces understanding, and behavioral changes introduced by provider-side model updates.

Problem

• Software Engineering agents
• LLM-based agents
• agent development challenges
• evaluation-driven development
• practitioner practices

Innovation

• Software Engineering Agents
• Evaluation-Driven Development
• Comprehension Debt
• LLM-Based Systems
• Mixed-Methods Study