Hint-Writing with Deferred AI Assistance: Fostering Critical Engagement in Data Science Education

2026-04-21

AI Summary

This study investigates the design of AI-assisted prompt-writing activities in data science education to balance learning effectiveness with students’ cognitive engagement. Focusing on a graduate-level course, it proposes three human-AI collaboration modes: independent writing, immediate AI assistance, and an innovative “delayed AI assistance” approach—wherein students first draft prompts autonomously and subsequently revise them using AI-generated feedback. A randomized controlled trial conducted in an authentic classroom setting reveals that delayed AI assistance significantly improves prompt quality, enhances students’ ability to identify diverse error types, and effectively fosters critical thinking and judicious use of AI tools. These findings offer a pedagogical design paradigm for AI-enhanced education that thoughtfully integrates learner autonomy with timely support.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: Prompt Engineering / Prompting
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Generating hints for incorrect code is a cognitively demanding task that fosters learning and metacognitive development. This study investigates three designs for personalized, scalable, and reflective hint-writing activities within a data science course: (i) writing a hint independently, (ii) writing a hint with on-demand AI assistance, and (iii) deferred AI assistance, in which students first write a hint independently and then revise it with the help of an AI-generated one. We examine how AI support can scaffold the learning process without diminishing students' productive cognitive effort. Through a randomized controlled experiment with graduate-level students (N=97), we found that deferring AI assistance leads to the highest-quality hints. Further, this design helps students identify a wide range of mistakes they otherwise struggle to identify without any AI assistance. Students valued these activities as opportunities to practice debugging and critically engage with AI outputs--skills that are now critical for learners to acquire as programming becomes increasingly automated and the use of AI for learning grows. Our findings also highlight key considerations for designing student-AI collaborative learning experiences to sustain student engagement, maintain appropriate cognitive load, and mitigate negative effects of AI, such as introducing redundancies and extraneous information into student work.

Problem

• hint-writing
• AI assistance
• critical engagement
• data science education
• cognitive effort

Innovation

• deferred AI assistance
• hint-writing
• critical engagement
• cognitive scaffolding
• AI-augmented learning