SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework

2026-06-30

AI Summary

This study addresses the absence of training and evaluation benchmarks grounded in authentic teacher writing feedback for large language models (LLMs). To bridge this gap, the authors introduce SEFORA, the first multi-genre, multi-round corpus of real classroom writing feedback, comprising students’ revised drafts alongside teachers’ line-by-line comments. They further propose UniMatch, an interpretable reference-based evaluation framework that quantifies the alignment between model-generated and teacher-provided feedback through semantic unit matching and optimal assignment algorithms. Evaluations across 74 experimental configurations reveal that even the strongest LLMs achieve only a 0.4 F1 score, with performance degrading as output length increases, indicating a fundamental difficulty in capturing the core aspects of teacher feedback.

Abstract

Effective writing feedback is among the strongest drivers of student learning, yet producing it at scale is labor-intensive. LLMs offer a natural path to scaling writing support, but two gaps stand in the way: few public corpora capture how instructors actually deliver feedback in real classrooms, and no reliable method measures whether generated feedback aligns with what an instructor would write. We address both. SEFORA is a public corpus pairing instructor inline feedback with assignment prompts, rubrics, scores, and multi-draft revisions across various college writing genres, comprising 564 drafts and 8,240 instructor annotations. UniMatch is a reference-based evaluation framework for open-ended generation: it segments feedback into feedback units, scores their semantic correspondence under instructor-derived criteria, and aligns them via optimal matching to yield interpretable precision, recall, and F1. Across 74 experimental configurations spanning multiple LLMs, no setting exceeds 0.4 F1. UniMatch reveals that models struggle to identify the feedback instructors would prioritize, and performance degrades as models generate more.

Problem

• writing feedback
• instructor feedback
• LLM evaluation
• feedback alignment
• educational corpus

Innovation

• feedback corpus
• LLM evaluation
• semantic matching
• writing feedback
• UniMatch