The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance

2026-05-16

AI Summary

This study addresses the challenge novice programmers face in debugging due to ambiguous compiler error messages, a problem exacerbated by the limited scalability of human feedback. The authors design three types of large language model (LLM)-generated feedback varying in guidance level and conduct an empirical study on an online programming platform, comparing them against a baseline that provides only raw compiler errors. Results demonstrate that LLM-generated feedback significantly reduces students’ problem-solving time, with low-guidance feedback yielding the strongest performance—an effect moderated by students’ prior programming experience. These findings challenge the common assumption that higher guidance inherently leads to better learning outcomes and offer novel insights for designing adaptive feedback mechanisms in programming education.

Abstract

When programming students encounter errors in their code, compiler messages or static analysis output often provide limited guidance, particularly for novice programmers. Personalized feedback from instructors can be effective but does not scale well. Recent advances in large language models (LLMs) enable automated feedback generation at scale. This study examines whether LLM-generated feedback with different levels of guidance is associated with differences in students' problem-solving behavior. We analyze effects on time to solution and number of attempts, and examine whether these effects differ by programming experience. We design three feedback types and compare them to a baseline in which students receive only compiler error messages. Results from an online programming course show that LLM-generated feedback is associated with faster time to solution compared to the no-feedback baseline, with less guided feedback showing slightly stronger effects. Overall, the findings suggest that feedback structure plays an important role in how students progress toward correct solutions and motivate further work on adaptive feedback designs and longer-term learning outcomes.

Problem

• LLM-generated feedback
• programming education
• automated feedback
• novice programmers
• error diagnosis

Innovation

• structured feedback
• large language models
• programming education
• adaptive feedback
• novice programmers