Auto-grader Feedback Utilization and Its Impacts: An Observational Study Across Five Community Colleges

2025-07-17

AI Summary

Prior research often conflates feedback provision with feedback effectiveness, overlooking how students’ actual engagement with automated feedback mediates learning outcomes. Method: Drawing on observational data from introductory Python courses across five U.S. community colleges, we integrated submission logs and fine-grained feedback access records to empirically distinguish feedback users from non-users. Contribution/Results: We find that students who frequently accessed auto-grader feedback achieved significantly higher assignment scores; moreover, submissions following feedback review showed substantially greater performance gains than those ignoring feedback. This study is the first to demonstrate—using behavioral usage data—that feedback utilization, not merely its availability, serves as a critical mediating variable in programming learning. Our findings challenge the implicit assumption that feedback delivery equates to feedback efficacy, providing robust empirical evidence to inform both the educational evaluation of auto-grading systems and the design of more effective, behaviorally grounded feedback mechanisms.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: User Experience and Usability
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Automated grading systems, or auto-graders, have become ubiquitous in programming education, and the way they generate feedback has become increasingly automated as well. However, there is insufficient evidence regarding auto-grader feedback's effectiveness in improving student learning outcomes, in a way that differentiates students who utilized the feedback and students who did not. In this study, we fill this critical gap. Specifically, we analyze students' interactions with auto-graders in an introductory Python programming course, offered at five community colleges in the United States. Our results show that students checking the feedback more frequently tend to get higher scores from their programming assignments overall. Our results also show that a submission that follows a student checking the feedback tends to receive a higher score than a submission that follows a student ignoring the feedback. Our results provide evidence on auto-grader feedback's effectiveness, encourage their increased utilization, and call for future work to continue their evaluation in this age of automation

Problem

• Assess auto-grader feedback's impact on student learning outcomes
• Compare feedback utilization versus non-utilization in programming courses
• Evaluate feedback effectiveness across five community colleges

Innovation

• Analyzes auto-grader feedback usage in Python courses
• Links feedback checks to higher assignment scores
• Compares outcomes between feedback users and non-users