How Adding Metacognitive Requirements in Support of AI Feedback in Practice Exams Transforms Student Learning Behaviors

2025-05-19

AI Summary

Personalized feedback is unsustainable in large-scale STEM courses. This study develops an AI-powered practical examination system for introductory biology, requiring students to articulate explanatory reasoning and self-assess confidence levels for each response; GPT-4o then generates tailored feedback and precise textbook references to scaffold metacognitive reflection. Our key contribution is the first empirical demonstration that structured metacognitive prompts—explanation plus confidence calibration—significantly outperform feedback complexity per se in promoting learning transfer. We observe substantial behavioral shifts: 40% of students proactively consulted recommended textbook sections (far exceeding typical engagement), 82.1% reported increased confidence on midterm topics, and 73.4% accurately recalled and applied specific concepts. User satisfaction reached 4.1/5. This work establishes the critical efficacy of lightweight, scaffolded metacognitive interventions in scaling AI-enhanced education.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Providing personalized, detailed feedback at scale in large undergraduate STEM courses remains a persistent challenge. We present an empirically evaluated practice exam system that integrates AI generated feedback with targeted textbook references, deployed in a large introductory biology course. Our system encourages metacognitive behavior by asking students to explain their answers and declare their confidence. It uses OpenAI's GPT-4o to generate personalized feedback based on this information, while directing them to relevant textbook sections. Through interaction logs from consenting participants across three midterms (541, 342, and 413 students respectively), totaling 28,313 question-student interactions across 146 learning objectives, along with 279 surveys and 23 interviews, we examined the system's impact on learning outcomes and engagement. Across all midterms, feedback types showed no statistically significant performance differences, though some trends suggested potential benefits. The most substantial impact came from the required confidence ratings and explanations, which students reported transferring to their actual exam strategies. About 40 percent of students engaged with textbook references when prompted by feedback -- far higher than traditional reading rates. Survey data revealed high satisfaction (mean rating 4.1 of 5), with 82.1 percent reporting increased confidence on practiced midterm topics, and 73.4 percent indicating they could recall and apply specific concepts. Our findings suggest that embedding structured reflection requirements may be more impactful than sophisticated feedback mechanisms.

Problem

• Addressing scalable personalized feedback in large STEM courses
• Encouraging metacognitive behaviors through AI-generated feedback
• Evaluating impact of confidence ratings on learning outcomes

Innovation

• AI-generated feedback with textbook references
• Metacognitive prompts for confidence and explanations
• Structured reflection requirements enhance learning