REFINE: Real-world Exploration of Interactive Feedback and Student Behaviour

2026-03-30

AI Summary

This work proposes a localized multi-agent feedback system based on small open-source large language models (LLMs), addressing the limitations of conventional feedback systems that treat feedback as static and unidirectional, thereby failing to support personalized, interactive formative guidance. The system uniquely implements fully interactive feedback through a multi-agent architecture featuring pedagogically oriented feedback generation, an LLM-as-a-judge regeneration mechanism guided by human-aligned criteria, and a context-aware reflexive tool-calling agent enabling students to ask follow-up questions and engage in dynamic dialogue. Experimental results demonstrate that criterion-guided regeneration significantly enhances feedback quality, and the interactive agents achieve performance comparable to state-of-the-art closed-source models in both efficiency and quality. Deployment in real classroom settings further validates the system’s effectiveness in guiding student inquiry.

Abstract

Formative feedback is central to effective learning, yet providing timely, individualised feedback at scale remains a persistent challenge. While recent work has explored the use of large language models (LLMs) to automate feedback, most existing systems still conceptualise feedback as a static, one-way artifact, offering limited support for interpretation, clarification, or follow-up. In this work, we introduce REFINE, a locally deployable, multi-agent feedback system built on small, open-source LLMs that treats feedback as an interactive process. REFINE combines a pedagogically-grounded feedback generation agent with an LLM-as-a-judge-guided regeneration loop using a human-aligned judge, and a self-reflective tool-calling interactive agent that supports student follow-up questions with context-aware, actionable responses. We evaluate REFINE through controlled experiments and an authentic classroom deployment in an undergraduate computer science course. Automatic evaluations show that judge-guided regeneration significantly improves feedback quality, and that the interactive agent produces efficient, high-quality responses comparable to a state-of-the-art closed-source model. Analysis of real student interactions further reveals distinct engagement patterns and indicates that system-generated feedback systematically steers subsequent student inquiry. Our findings demonstrate the feasibility and effectiveness of multi-agent, tool-augmented feedback systems for scalable, interactive feedback.

Problem

• formative feedback
• interactive feedback
• student behaviour
• large language models
• scalable feedback

Innovation

• interactive feedback
• multi-agent LLM system
• judge-guided regeneration
• tool-augmented interaction
• formative assessment