How Students Use AI Feedback Matters: Experimental Evidence on Physics Achievement and Autonomy

2025-05-13

AI Summary

The educational impact of generative artificial intelligence (GAI) for personalized feedback in physics learning remains poorly understood, particularly regarding how usage patterns—not just technological capability—moderate outcomes. Method: Two randomized controlled trials (N = 387) compared mandatory versus autonomous, on-demand GAI use, integrating multidimensional autonomy scales and objective academic performance measures. Contribution/Results: Mandatory use significantly improved achievement among low-performing students (d = 0.673) but impaired performance and autonomy among medium- and high-performing students (d = −0.539, −0.477). Conversely, autonomous use enhanced achievement for high performers (d = 0.378) yet undermined self-regulation in low performers (d = −0.383). This study provides the first empirical evidence of pronounced heterogeneity in AI-mediated learning effects, demonstrating that pedagogical value hinges not on technology deployment per se, but on human–AI collaboration frameworks explicitly designed to align with learners’ competence levels and agentic capacities.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Intelligent Robots: Manipulation
• Philosophy and Ethics of AI: Artificial General Intelligence

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Despite the precision and adaptiveness of generative AI (GAI)-powered feedback provided to students, existing practice and literature might ignore how usage patterns impact student learning. This study examines the heterogeneous effects of GAI-powered personalized feedback on high school students' physics achievement and autonomy through two randomized controlled trials, with a major focus on usage patterns. Each experiment lasted for five weeks, involving a total of 387 students. Experiment 1 (n = 121) assessed compulsory usage of the personalized recommendation system, revealing that low-achieving students significantly improved academic performance (d = 0.673, p<0.05) when receiving AI-generated heuristic solution hints, whereas medium-achieving students' performance declined (d = -0.539, p<0.05) with conventional answers provided by workbook. Notably, high-achieving students experienced a significant decline in self-regulated learning (d = -0.477, p<0.05) without any significant gains in achievement. Experiment 2 (n = 266) investigated the usage pattern of autonomous on-demand help, demonstrating that fully learner-controlled AI feedback significantly enhanced academic performance for high-achieving students (d = 0.378, p<0.05) without negatively impacting their autonomy. However, autonomy notably declined among lower achievers exposed to on-demand AI interventions (d = -0.383, p<0.05), particularly in the technical-psychological dimension (d = -0.549, p<0.05), which has a large overlap with self-regulation. These findings underscore the importance of usage patterns when applying GAI-powered personalized feedback to students.

Problem

• Examining how AI feedback usage patterns affect student learning.
• Assessing impact of AI feedback on physics achievement and autonomy.
• Investigating learner-controlled vs compulsory AI feedback effects.

Innovation

• GAI-powered personalized feedback for students
• Randomized controlled trials on usage patterns
• Autonomous on-demand AI feedback enhances performance