Problem
Research questions and friction points this paper is trying to address.
AI code completion
critical engagement
behavioral signals
attention checks
reflective engagement
To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks
🤖 AI Summary
This study addresses the tendency of students to uncritically accept AI-generated code suggestions during programming tasks. To this end, it introduces Clover—the first analytical framework that integrates behavioral logging with attention-check mechanisms to quantify reflective engagement in AI-assisted programming. By tracking interaction patterns such as acceptance methods (e.g., one-click Tab acceptance) and dwell time on suggested code, Clover constructs a set of metrics that capture learners’ attentiveness and critical evaluation behaviors. The findings reveal a significant negative correlation between frequent use of immediate acceptance (via Tab) and attentional performance, whereas longer code inspection durations are positively associated with higher attention levels. This work provides an empirical and methodological foundation for measuring and fostering more deliberate and reflective interactions between learners and AI programming tools.
📝 Abstract
AI code completion tools, such as Github Copilot, provide students with code suggestions to help them write programs. However, recent qualitative studies suggest that students fail to critically evaluate these suggestions. We present Clover, a code completion tool that logs students' interactions with code suggestions and additionally offers attention checks to probe reflective engagement during programming tasks. We also develop a taxonomy of behavioral interaction metrics for AI-assisted programming, informed by literature. We analyzed relationships between interaction patterns, engagement with attention checks, and task performance. We observed that higher rates of tab accept were associated with lower attention check performance, while increased dwell time was associated with higher attention check performance. We conclude by discussing how programming process data and attention checks might support reflective engagement in AI-assisted programming.
Innovation
Methods, ideas, or system contributions that make the work stand out.
AI code completion
attention checks
behavioral signals
reflective engagement
interaction metrics
🔎 Similar Papers
No similar papers found.
