SWE-Together: Evaluating Coding Agents in Interactive User Sessions

2026-06-29

AI Summary

This work addresses the limitations of existing code agent evaluations, which are predominantly confined to static, single-turn tasks and fail to capture the dynamic nature of real-world human-agent collaboration involving multi-turn interactions, goal clarification, and error correction. To bridge this gap, we propose SWE-Together—the first multi-turn evaluation framework tailored for realistic interactive scenarios—reconstructing 109 verifiable repository-level tasks from 11,260 authentic developer conversations. The framework features a repository state tracker with full reproducibility, an LLM-driven reactive user simulator that faithfully preserves original user intent, and a composite metric integrating final correctness with the frequency of user interventions. Experimental results demonstrate that state-of-the-art agents evaluated on this benchmark not only achieve higher success rates but also require fewer user interventions, significantly enhancing the realism and quality of collaborative coding experiences.

Abstract

Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.

Problem

• coding agents
• interactive evaluation
• multi-turn benchmark
• user-agent interaction
• code generation

Innovation

• multi-turn benchmark
• interactive coding agent evaluation
• user simulator
• repository-level tasks
• corrective feedback turns