Examining the Use and Impact of an AI Code Assistant on Developer Productivity and Experience in the Enterprise

2024-12-09

arXiv.org

AI Summary

Despite growing adoption of large language model (LLM)-based code assistants in industry, empirical evidence on their differential impact on developer productivity and experience in real-world enterprise settings remains scarce. Method: This study conducts the first systematic, mixed-methods evaluation of IBM watsonx Code Assistant (WCA) within an enterprise environment, integrating 669 survey responses, 15 unmoderated usability sessions, and qualitative interviews to examine usage motivations, quality expectations, and responsibility attribution. Contribution/Results: WCA yields a net positive productivity gain overall, yet benefits are highly heterogeneous—approximately 30% of developers report no significant improvement. Substantial divergences emerge in perceptions of code ownership, debugging overhead, and trust thresholds. The study reveals that LLM-assisted programming delivers non-uniform gains across developer cohorts and introduces novel governance challenges around accountability, liability, and human-AI collaboration. These findings provide empirically grounded insights for integrating AI into software engineering practice and inform responsible deployment strategies for enterprise-grade coding assistants.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: User Experience and Usability
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

AI assistants are being created to help software engineers conduct a variety of coding-related tasks, such as writing, documenting, and testing code. We describe the use of the watsonx Code Assistant (WCA), an LLM-powered coding assistant deployed internally within IBM. Through surveys of two user cohorts (N=669) and unmoderated usability testing (N=15), we examined developers' experiences with WCA and its impact on their productivity. We learned about their motivations for using (or not using) WCA, we examined their expectations of its speed and quality, and we identified new considerations regarding ownership of and responsibility for generated code. Our case study characterizes the impact of an LLM-powered assistant on developers' perceptions of productivity and it shows that although such tools do often provide net productivity increases, these benefits may not always be experienced by all users.

Problem

• Impact of AI code assistant on developer productivity
• User experience with LLM-powered coding tools
• Ownership and responsibility for AI-generated code

Innovation

• LLM-powered coding assistant enhances productivity
• Surveys and usability testing evaluate developer experience
• Identifies ownership and responsibility for AI-generated code