Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows

2025-07-10

AI Summary

This study investigates the impact of highly autonomous AI coding agents (e.g., OpenHands) versus conventional programming assistants (e.g., GitHub Copilot) on developer workflows, efficiency, and experience. Method: Employing a human factors experimental design, we conduct the first empirical comparison of these two paradigms in realistic development settings, integrating behavioral telemetry, task performance metrics, and subjective user feedback for multidimensional analysis. Results: Coding agents significantly reduce user interaction overhead and enable end-to-end task automation, thereby expanding the scope of tasks delegable to AI. However, they introduce adoption barriers—including intent opacity and behavioral unpredictability. Contribution: We empirically demonstrate that increased automation precipitates a fundamental shift in human–AI collaboration paradigms and derive evidence-based interaction design principles specifically for highly autonomous AI systems.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

Developers now have access to a growing array of increasingly autonomous AI tools to support software development. While numerous studies have examined developer use of copilots, which can provide chat assistance or code completions, evaluations of coding agents, which can automatically write files and run code, still largely rely on static benchmarks without humans-in-the-loop. In this work, we conduct the first academic study to explore developer interactions with coding agents and characterize how more autonomous AI tools affect user productivity and experience, compared to existing copilots. We evaluate two leading copilot and agentic coding assistants, GitHub Copilot and OpenHands, recruiting participants who regularly use the former. Our results show agents have the potential to assist developers in ways that surpass copilots (e.g., completing tasks that humans might not have accomplished before) and reduce the user effort required to complete tasks. However, there are challenges involved in enabling their broader adoption, including how to ensure users have an adequate understanding of agent behaviors. Our results not only provide insights into how developer workflows change as a result of coding agents but also highlight how user interactions with agents differ from those with existing copilots, motivating a set of recommendations for researchers building new agents. Given the broad set of developers who still largely rely on copilot-like systems, our work highlights key challenges of adopting more agentic systems into developer workflows.

Problem

• How autonomous AI coding agents transform developer workflows
• Comparing productivity impacts of coding agents versus copilots
• Challenges in adopting agentic systems for developers

Innovation

• Study developer interactions with autonomous coding agents
• Compare GitHub Copilot and OpenHands assistants
• Identify challenges in adopting agentic systems