AI-Mediated Feedback Improves Student Revisions: A Randomized Trial with FeedbackWriter in a Large Undergraduate Course

2026-02-18

AI Summary

This study addresses the lack of empirical evidence comparing the effectiveness of AI-generated versus human-provided feedback in student writing revision. Conducted as a randomized controlled trial in a large undergraduate economics course, it deployed FeedbackWriter—a large language model–based system that offers teaching assistants knowledge-intensive, rubric-aligned feedback suggestions, which they could accept, modify, or disregard. Results show that students receiving AI-assisted feedback produced significantly higher-quality revisions, with the degree of teaching assistant adoption positively correlated with learning gains. This work provides the first empirical validation, supported by data from 1,366 essays, that AI-augmented teaching assistants can effectively enhance the quality of student writing revisions in authentic instructional settings.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Intelligent Robots: Embodied AI
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Despite growing interest in using LLMs to generate feedback on students' writing, little is known about how students respond to AI-mediated versus human-provided feedback. We address this gap through a randomized controlled trial in a large introductory economics course (N=354), where we introduce and deploy FeedbackWriter - a system that generates AI suggestions to teaching assistants (TAs) while they provide feedback on students' knowledge-intensive essays. TAs have the full capacity to adopt, edit, or dismiss the suggestions. Students were randomly assigned to receive either handwritten feedback from TAs (baseline) or AI-mediated feedback where TAs received suggestions from FeedbackWriter. Students revise their drafts based on the feedback, which is further graded. In total, 1,366 essays were graded using the system. We found that students receiving AI-mediated feedback produced significantly higher-quality revisions, with gains increasing as TAs adopted more AI suggestions. TAs found the AI suggestions useful for spotting gaps and clarifying rubrics.

Problem

• AI-mediated feedback
• student revisions
• LLM-generated feedback
• human feedback
• writing improvement

Innovation

• AI-mediated feedback
• human-in-the-loop
• large language models
• writing revision
• randomized controlled trial