Autonomous LLM-generated Feedback for Student Exercises in Introductory Software Engineering Courses

2026-04-22

AI Summary

This study addresses the challenge of providing personalized feedback in introductory software engineering courses, where large and diverse student cohorts coupled with high student-to-instructor ratios often impede timely and individualized assessment. To tackle this issue, the authors propose NAILA—the first fully automated, scalable AI feedback system tailored for software engineering education. Built upon large language models (LLMs), NAILA leverages instructor-defined reference solutions and specialized prompting templates to automatically parse student submissions in open document formats and deliver round-the-clock personalized feedback. Empirical evaluation involving over 900 students demonstrates that NAILA is highly accepted and consistently utilized, significantly enhancing students’ perceived learning progress and actual academic performance, thereby validating the efficacy and feasibility of LLMs in delivering feedback within programming education contexts.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Introductory Software Engineering (SE) courses face rapidly increasing student enrollment numbers, participants with diverse backgrounds and the influence of Generative AI (GenAI) solutions. High teacher-to-student ratios often challenge providing timely, high-quality, and personalized feedback a significant challenge for educators. To address these challenges, we introduce NAILA, a tool that provides 24/7 autonomous feedback for student exercises. Utilizing GenAI in the form of modern LLMs, NAILA processes student solutions provided in open document formats, evaluating them against teacher-defined model solutions through specialized prompt templates. We conducted an empirical study involving 900+ active students at the University of Duisburg-Essen to assess four main research questions investigating (1) the underlying motivations that drive students to either adopt or reject NAILA, (2) user acceptance by measuring perceived usefulness and ease of use alongside subjective learning progress, (3) how often and how consistently students engage with NAILA, and (4) how using NAILA to receive AI feedback impacts on academic performance compared to human feedback.

Problem

• student feedback
• software engineering education
• generative AI
• large language models
• personalized learning

Innovation

• Autonomous Feedback
• Generative AI
• Large Language Models
• Software Engineering Education
• Prompt Engineering