Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

2025-07-11

AI Summary

This study investigates the causal impact of AI programming tools on the productivity of experienced open-source developers in early 2025. Method: We conduct the first randomized controlled trial (RCT) embedded in mature, real-world open-source projects, comparing an intervention group using Cursor Pro with Claude 3.5/3.7 Sonnet against a control group. Primary outcome is task completion time; secondary measures include developer self-assessments and expert predictions for validation. Contribution/Results: Contrary to developers’ expectation of a 24% speedup, AI assistance significantly *increased* mean task time by 19% in high-maturity projects—a negative effect robust across multiple sensitivity and specification tests. This is the first RCT to empirically reveal a counterintuitive inverse relationship between AI assistance and expert developer efficiency in authentic open-source settings. We systematically identify and evaluate 20 potential moderating factors, providing critical empirical evidence on the boundary conditions under which AI augments—or impedes—software development productivity.

Technology Category

• Humans and AI: User Experience and Usability
• Reasoning under Uncertainty: Causality
• Intelligent Robots: Embodied AI

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. Each task is randomly assigned to allow or disallow usage of early 2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). To understand this result, we collect and evaluate evidence for 20 properties of our setting that a priori could contribute to the observed slowdown effect--for example, the size and quality standards of projects, or prior developer experience with AI tooling. Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.

Problem

• Measure AI tool impact on open-source developer productivity
• Compare developer forecasts with actual AI tool effects
• Investigate unexpected productivity slowdown with AI tools

Innovation

• Randomized controlled trial (RCT) design
• Cursor Pro and Claude 3.5/3.7 Sonnet tools
• Evaluated 20 properties for slowdown effect