A Survey on Feedback Types in Automated Programming Assessment Systems

2025-10-21

AI Summary

Existing Automated Programming Assessment Systems (APAS) rely solely on predefined unit tests, yielding narrow, non-personalized feedback that inadequately supports learning. Method: This study comparatively evaluates three feedback modalities—compiler output, unit test results, and AI-generated feedback from large language models (LLMs)—and proposes a hybrid “unit test + AI-generated” feedback mechanism. A large-scale user study—including both quantitative performance metrics and qualitative survey data—was conducted to assess efficacy. Contribution/Results: While students subjectively preferred unit-test feedback, the AI-only group demonstrated significantly superior problem-solving performance (p < 0.01). The hybrid approach synergistically combines the precision of unit tests with the semantic richness and explanatory depth of LLMs, enhancing both feedback quality and learning outcomes. This work provides empirical evidence and a novel paradigm for developing personalized, high-utility automated feedback systems in programming education.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

With the recent rapid increase in digitization across all major industries, acquiring programming skills has increased the demand for introductory programming courses. This has further resulted in universities integrating programming courses into a wide range of curricula, including not only technical studies but also business and management fields of study. Consequently, additional resources are needed for teaching, grading, and tutoring students with diverse educational backgrounds and skills. As part of this, Automated Programming Assessment Systems (APASs) have emerged, providing scalable and high-quality assessment systems with efficient evaluation and instant feedback. Commonly, APASs heavily rely on predefined unit tests for generating feedback, often limiting the scope and level of detail of feedback that can be provided to students. With the rise of Large Language Models (LLMs) in recent years, new opportunities have emerged as these technologies can enhance feedback quality and personalization. To investigate how different feedback mechanisms in APASs are perceived by students, and how effective they are in supporting problem-solving, we have conducted a large-scale study with over 200 students from two different universities. Specifically, we compare baseline Compiler Feedback, standard Unit Test Feedback, and advanced LLM-based Feedback regarding perceived quality and impact on student performance. Results indicate that while students rate unit test feedback as the most helpful, AI-generated feedback leads to significantly better performances. These findings suggest combining unit tests and AI-driven guidance to optimize automated feedback mechanisms and improve learning outcomes in programming education.

Problem

• Investigating student perceptions of automated programming feedback mechanisms
• Comparing effectiveness of compiler, unit test and AI-generated feedback types
• Optimizing automated assessment systems by combining unit tests with AI guidance

Innovation

• Combining unit tests with AI-driven feedback mechanisms
• Leveraging Large Language Models for personalized feedback
• Optimizing automated feedback to enhance learning outcomes