Partnering with AI: A Pedagogical Feedback System for LLM Integration into Programming Education

2025-06-30

AI Summary

This study addresses the gap in existing programming education where LLM-generated feedback neglects foundational pedagogical principles—such as mastery learning and adaptive pacing. We propose the first LLM feedback framework explicitly grounded in educational theory and informed by teacher practice. Methodologically, we design a web-based, human-AI collaborative feedback system that integrates large language models with a hybrid assessment architecture, enabling bidirectional validation of feedback quality and pedagogical effectiveness. Our contributions are threefold: (1) the first formal integration of mastery learning, dynamic adaptation, and other evidence-based instructional principles into LLM prompting strategies and feedback logic; (2) empirical evidence demonstrating that the system delivers timely, accurate feedback—outperforming human feedback in certain scenarios—and receiving strong endorsement from practicing educators; and (3) validation that LLMs must operate in close collaboration with teachers to address the complexity and dynamism inherent in real-world classroom settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-in-the-loop Machine Learning
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Feedback is one of the most crucial components to facilitate effective learning. With the rise of large language models (LLMs) in recent years, research in programming education has increasingly focused on automated feedback generation to help teachers provide timely support to every student. However, prior studies often overlook key pedagogical principles, such as mastery and progress adaptation, that shape effective feedback strategies. This paper introduces a novel pedagogical framework for LLM-driven feedback generation derived from established feedback models and local insights from secondary school teachers. To evaluate this framework, we implemented a web-based application for Python programming with LLM-based feedback that follows the framework and conducted a mixed-method evaluation with eight secondary-school computer science teachers. Our findings suggest that teachers consider that, when aligned with the framework, LLMs can effectively support students and even outperform human teachers in certain scenarios through instant and precise feedback. However, we also found several limitations, such as its inability to adapt feedback to dynamic classroom contexts. Such a limitation highlights the need to complement LLM-generated feedback with human expertise to ensure effective student learning. This work demonstrates an effective way to use LLMs for feedback while adhering to pedagogical standards and highlights important considerations for future systems.

Problem

• Developing pedagogical feedback system for LLM in programming education
• Addressing lack of mastery and progress adaptation in feedback
• Balancing LLM feedback with human expertise for effective learning

Innovation

• Pedagogical framework for LLM-driven feedback generation
• Web-based Python app with LLM feedback integration
• Mixed-method evaluation with secondary school teachers