Reflection-Satisfaction Tradeoff: Investigating Impact of Reflection on Student Engagement with AI-Generated Programming Hints

2025-12-04

AI Summary

This study investigates how AI prompt design in programming education can simultaneously foster student autonomy and deep learning. A randomized controlled trial was conducted within an online programming course to systematically compare the effects of prompt timing (pre- vs. post-task), self-regulation phase focus (planning, monitoring, or evaluation), and guidance type (directive vs. open-ended) on reflection quality and learner experience. Results indicate that pre-task prompts, emphasis on the planning phase, and directive guidance significantly enhance reflection depth—but concurrently reduce learners’ satisfaction with AI prompts; no significant differences emerged in immediate programming performance. These findings reveal a trade-off between reflection quality and prompt satisfaction, challenging the prevailing educational AI evaluation paradigm that prioritizes user satisfaction as a sole metric. The study advocates for an alternative paradigm wherein educational AI systems prioritize supporting learner autonomy and cognitive engagement over optimizing short-term experiential metrics.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Generative AI tools, such as AI-generated hints, are increasingly integrated into programming education to offer timely, personalized support. However, little is known about how to effectively leverage these hints while ensuring autonomous and meaningful learning. One promising approach involves pairing AI-generated hints with reflection prompts, asking students to review and analyze their learning, when they request hints. This study investigates the interplay between AI-generated hints and different designs of reflection prompts in an online introductory programming course. We conducted a two-trial field experiment. In Trial 1, students were randomly assigned to receive prompts either before or after receiving hints, or no prompt at all. Each prompt also targeted one of three SRL phases: planning, monitoring, and evaluation. In Trial 2, we examined two types of prompt guidance: directed (offering more explicit and structured guidance) and open (offering more general and less constrained guidance). Findings show that students in the before-hint (RQ1), planning (RQ2), and directed (RQ3) prompt groups produced higher-quality reflections but reported lower satisfaction with AI-generated hints than those in other conditions. Immediate performance did not differ across conditions. This negative relationship between reflection quality and hint satisfaction aligns with previous work on student mental effort and satisfaction. Our results highlight the need to reconsider how AI models are trained and evaluated for education, as prioritizing user satisfaction can undermine deeper learning.

Problem

• Investigating reflection's impact on student engagement with AI hints
• Exploring tradeoff between reflection quality and hint satisfaction
• Examining prompt design effects on learning in programming education

Innovation

• Pairing AI hints with reflection prompts
• Testing prompt timing and SRL phase targeting
• Comparing directed vs open prompt guidance