The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants

2024-08-07

AI Summary

This study investigates how AI-assisted tools influence developers’ information needs, information-seeking behaviors, productivity, and skill development. Method: A mixed-methods approach was employed—including in-depth interviews, surveys, and behavioral log analysis—across professional software developers. Contribution/Results: Results reveal that 32% of development time is spent on information seeking; while AI significantly improves information retrieval efficiency, it concurrently impedes deep learning. This work presents the first systematic characterization of the evolution of information-seeking behavior under AI mediation. It introduces the “foundational knowledge anchoring” theory: developers’ prior programming knowledge critically moderates both the effectiveness of AI-generated outputs and the quality of learning—stronger foundational knowledge yields superior AI-augmented learning outcomes. These findings provide empirically grounded design principles and cognitive mechanism explanations for human-AI co-design in developer tooling.

Technology Category

• Humans and AI: Other Foundations of Human Computation & AI
• Search and Optimization: Other Foundations of Search & Optimization
• Philosophy and Ethics of AI: Philosophical Foundations of AI

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Economic aspects and information design of Web, GenAI, and cloud computing

Abstract

About 32% of a software practitioners' day involves seeking and using information to support task completion. Although the information needs of software practitioners have been studied extensively, the impact of AI-assisted tools on their needs and information-seeking behaviors remains largely unexplored. To addresses this gap, we conducted a mixed-method study to understand AI-assisted information seeking behavior of practitioners and its impact on their perceived productivity and skill development. We found that developers are increasingly using AI tools to support their information seeking, citing increased efficiency as a key benefit. Our findings also amplify caveats that come with effectively using AI tools for information seeking, especially for learning and skill development, such as the importance of foundational developer knowledge that can guide and inform the information provided by AI tools. Our efforts have implications for the effective integration of AI tools into developer workflows as information retrieval systems and learning aids.

Problem

• Impact of AI tools on developers' information-seeking behaviors
• Effect of AI assistance on productivity and skill development
• Challenges in using AI tools for learning and information retrieval

Innovation

• Mixed-method study on AI-assisted information seeking
• AI tools enhance efficiency in developer workflows
• Foundational knowledge crucial for effective AI tool use