Towards Decoding Developer Cognition in the Age of AI Assistants

2025-01-05

AI Summary

This study investigates the empirical impact of AI programming assistants on cognitive load, programming efficiency, and self-efficacy across developer expertise levels (experts vs. novices). We employ a multimodal methodology: synchronized acquisition of EEG (to assess deep cognitive load), eye-tracking, interaction logs, NASA-TLX subjective workload ratings, and a controlled experimental design. For the first time, we integrate physiological signals with behavioral data to reveal the “efficiency paradox” of AI assistance: while experts leverage AI more efficiently and reduce certain cognitive demands, their perceived productivity gains are often inflated; critically, EEG evidence shows no significant attenuation of higher-order cognitive load. Our core contributions are: (1) establishing an objective, multimodal evaluation framework for AI programming efficacy; (2) empirically demonstrating that developer expertise is a key moderating variable—significantly shaping AI’s actual utility; and (3) challenging the prevailing subjectivity-biased “efficiency consensus” in current AI-assisted programming research.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Humans and AI: User Experience and Usability
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Background: The increasing adoption of AI assistants in programming has led to numerous studies exploring their benefits. While developers consistently report significant productivity gains from these tools, empirical measurements often show more modest improvements. While prior research has documented self-reported experiences with AI-assisted programming tools, little to no work has been done to understand their usage patterns and the actual cognitive load imposed in practice. Objective: In this exploratory study, we aim to investigate the role AI assistants play in developer productivity. Specifically, we are interested in how developers' expertise levels influence their AI usage patterns, and how these patterns impact their actual cognitive load and productivity during development tasks. We also seek to better understand how this relates to their perceived productivity. Method: We propose a controlled observational study combining physiological measurements (EEG and eye tracking) with interaction data to examine developers' use of AI-assisted programming tools. We will recruit professional developers to complete programming tasks both with and without AI assistance while measuring their cognitive load and task completion time. Through pre- and post-task questionnaires, we will collect data on perceived productivity and cognitive load using NASA-TLX.

Problem

• Artificial Intelligence Assistants
• Programming Performance
• Expertise Level

Innovation

• Brain Activity
• Eye-Tracking
• AI-assisted Programming