Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code

2026-06-24

AI Summary

This study investigates the gap between developers’ actual behavior and recommended practices when reviewing code explicitly labeled as generated by large language models (LLMs). Through a Wizard-of-Oz experiment integrating eye-tracking, Bayesian data analysis, and exit interviews, the research reveals— for the first time—that while LLM labeling does not enhance review depth, it significantly increases fixation duration. Moreover, developers dynamically adapt their review strategies based on the provided prompts. These findings bridge the divide between developers’ stated intentions and their observed behaviors, offering empirical evidence and actionable insights for the design of LLM-assisted programming tools and the formulation of organizational AI governance policies.

Abstract

Modern software development increasingly involves the use of large language models (LLMs) to generate code. Despite their rapid advancement, LLMs remain prone to errors and hallucinations, emphasizing the importance of careful code inspection. However, in practice, developers' trust in LLM-generated code and their willingness to review it thoroughly may differ from these recommendations. How developers actually behave when reviewing LLM-generated code remains largely unexplored. In this study, we conduct a Wizard-of-Oz experiment to examine how software engineers behave when code is explicitly labeled as LLM-generated during a code review task. We collect both behavioral data and participant feedback through eye-tracking and exit interviews. Combining Bayesian data analysis with qualitative analysis, we found that while the thoroughness of code review did not change for participants, they spent more time fixating on LLM-labelled code, indicating that the label itself influences attention. Practitioners also adapted their review strategy for LLM-labelled code by assessing the code based on specific criteria (e.g., logical correctness), or using the prompt to guide their review. These findings inform LLM-based tool design on labelling while incorporating the prompt as a software artifact. Our study reveals a gap between reviewers' intentions and actual reviewing behaviour, highlighting the need for software companies to revisit their AI policies (particularly regarding LLM-assisted development) to better support developers in reviewing LLM-generated code.

Problem

• code review
• large language models
• developer behavior
• LLM-generated code
• eye tracking

Innovation

• eye tracking
• LLM-generated code
• code review
• Wizard-of-Oz experiment
• prompt as artifact