Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

2026-06-11

AI Summary

Existing benchmarks for AI programming agents overlook their ability to collaborate with users in real-world scenarios. This work proposes the first automated evaluation framework tailored for conversational programming agents, introducing a role-based user simulator and an automatic dialogue quality assessment mechanism to comprehensively measure an agent’s capacity to solve practical coding tasks through interaction. The study reveals a misalignment between coding proficiency and conversational competence and addresses this gap by designing a schema-guided agent architecture to enhance interactive performance. Experimental results demonstrate consistent improvements of 3%–14% over multiple strong baselines, underscoring the importance of treating conversational ability as an independent and critical evaluation dimension in programming agent assessment.

Abstract

AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue SWE-Bench, an automatic benchmark dataset for evaluating the ability of coding agents to resolve real-world software engineering problems through dialogue with a user. We design a novel, persona-grounded user simulator to support our task evaluation, and augment our task evaluation with automatic evaluations of dialogue quality. We also propose a new schema-guided agent, aimed at improving the dialogue capabilities of off-the-shelf coding agents, which improves over strong baselines by 3-14%. Our results indicate that better coding models do not always correspond to better dialogue models, suggesting that dialogue capability is a distinct and currently understudied dimension of coding agent performance.

Problem

• dialogue-driven coding agents
• software engineering benchmark
• interactive coding assistants
• dialogue evaluation
• user simulator

Innovation

• Dialogue SWE-Bench
• persona-grounded user simulator
• schema-guided agent
• dialogue quality evaluation
• coding agent benchmarking