ChatGPT: Friend or Foe When Comprehending and Changing Unfamiliar Code

2026-05-11

AI Summary

This study investigates how AI influences developers’ cognitive processes and problem-solving behaviors when comprehending and modifying unfamiliar code, with particular focus on the mechanisms underlying entry into and escape from impasses. Grounded in Pólya’s four-stage problem-solving model and employing 25 inductively derived codes, the research leverages a controlled experiment and multi-source data triangulation—including think-aloud protocols, code changes, web searches, and LLM prompts—to systematically reveal, for the first time at the cognitive-behavioral level, AI’s impact across the entire programming workflow. The study identifies seven recurrent types of impasses and their bidirectional interaction patterns with AI, finding that although 90% of developers encounter impasses and frequently offload tasks to AI, their core problem-solving strategies remain largely unchanged; AI facilitates escape in certain contexts yet exacerbates difficulties in others.

Abstract

A rapidly growing body of research is examining how LLMs influence developers when they code. To date, this research has tended to focus on productivity and code quality outcomes, rather than the underlying cognitive processes involved in programming. To address this gap, we report on the results of an exploratory laboratory study of ten advanced student developers (five with support from AI and five without) who had to make a non-trivial extension to a sizable software system. Leveraging Polya's four problem-solving phases and 25 inductively-generated codes detailing distinct problem-solving behaviors as the primary lenses, we examined: (1) how AI impacted the problem-solving approach the developers used to solve the programming task, and (2) how AI impacted their progress when they became stuck. For the analysis, we triangulated data across multiple sources (e.g., think-aloud, code changes, web searches, and LLM prompts). Unexpectedly, while developers in the AI group repeatedly turned to the AI tool to offload certain aspects of the process, all detailed problem-solving behaviors appeared in both groups. We also found that nine out of ten participants found themselves stuck in their work, but with key differences in how they became stuck and unstuck. We highlight seven distinct causes for being stuck and highlight how AI in some cases helped and in other cases hindered becoming unstuck.

Problem

• large language models
• programming cognition
• problem-solving
• developer productivity
• code comprehension

Innovation

• cognitive processes
• problem-solving behaviors
• large language models
• developer productivity
• code comprehension