SWE-chat: Coding Agent Interactions From Real Users in the Wild

2026-04-22

AI Summary

This study addresses the lack of empirical analysis on how developers actually use AI programming agents in real-world settings and the practical utility of their outputs. The authors construct and continuously update the first large-scale, automatically collected live dataset of real-world interactions with programming agents, capturing user prompts, tool invocations, and code submissions. The dataset integrates automated log parsing, fine-grained code provenance tracking, security vulnerability detection, and user feedback identification. Their findings reveal that while AI agents lead coding in 41% of sessions, only 44% of the agent-generated code is ultimately retained in final submissions—and such code is more prone to introducing security vulnerabilities. Moreover, users actively correct or interrupt agent outputs in 44% of interaction turns, highlighting both the current limitations of AI programming agents and emerging patterns of human-agent collaboration.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Interaction Techniques and Devices
• Multiagent Systems: Adversarial Agents

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Agentic search

Abstract

AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild. The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls. SWE-chat is a living dataset; our collection pipeline automatically and continually discovers and processes sessions from public repositories. Leveraging SWE-chat, we provide an initial empirical characterization of real-world coding agent usage and failure modes. We find that coding patterns are bimodal: in 41% of sessions, agents author virtually all committed code ("vibe coding"), while in 23%, humans write all code themselves. Despite rapidly improving capabilities, coding agents remain inefficient in natural settings. Just 44% of all agent-produced code survives into user commits, and agent-written code introduces more security vulnerabilities than code authored by humans. Furthermore, users push back against agent outputs -- through corrections, failure reports, and interruptions -- in 44% of all turns. By capturing complete interaction traces with human vs. agent code authorship attribution, SWE-chat provides an empirical foundation for moving beyond curated benchmarks towards an evidence-based understanding of how AI agents perform in real developer workflows.

Problem

• AI coding agents
• real-world usage
• empirical evidence
• developer workflows
• code authorship

Innovation

• SWE-chat
• real-world coding agent interactions
• human-AI collaboration
• code authorship attribution
• empirical dataset