AUTOMATIC FEEDBACK GENERATION FOR SHORT ANSWER QUESTIONS USING ANSWER DIAGNOSTIC GRAPHS

2024-07-01

EDULEARN Proceedings

AI Summary

Existing automated feedback systems struggle to deeply diagnose logical and structural flaws in students’ short-answer responses to reading comprehension questions, and often lack both specificity and pedagogical efficacy for improvement. This paper addresses short-answer tasks in expository text comprehension by proposing the first automated feedback framework tailored to text-dependent questions. It introduces the Answer Diagnosis Graph (ADG)—a novel modeling approach that explicitly integrates textual logical structure, rubric criteria, and error taxonomies. The framework leverages dependency parsing, semantic role labeling, graph-based structural modeling, and template-driven natural language generation to produce personalized, actionable feedback. In an empirical study with 39 Japanese high school students, the system significantly enhanced participants’ error identification accuracy and intrinsic motivation. Results empirically validate the educational value and effectiveness of structured diagnostic feedback in fostering metacognitive development.

Technology Category

• Natural Language Processing: Question Answering
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Community question answering
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Short-reading comprehension questions help students understand text structure but lack effective feedback. Students struggle to identify and correct errors, while manual feedback creation is labor-intensive. This highlights the need for automated feedback linking responses to a scoring rubric for deeper comprehension. Despite advances in Natural Language Processing (NLP), research has focused on automatic grading, with limited work on feedback generation. To address this, we propose a system that generates feedback for student responses. Our contributions are twofold. First, we introduce the first system for feedback on short-answer reading comprehension. These answers are derived from the text, requiring structural understanding. We propose an"answer diagnosis graph,"integrating the text's logical structure with feedback templates. Using this graph and NLP techniques, we estimate students' comprehension and generate targeted feedback. Second, we evaluate our feedback through an experiment with Japanese high school students (n=39). They answered two 70-80 word questions and were divided into two groups with minimal academic differences. One received a model answer, the other system-generated feedback. Both re-answered the questions, and we compared score changes. A questionnaire assessed perceptions and motivation. Results showed no significant score improvement between groups, but system-generated feedback helped students identify errors and key points in the text. It also significantly increased motivation. However, further refinement is needed to enhance text structure understanding.

Problem

• Automatic Feedback Systems
• Depth of Answers
• Self-correction Ability

Innovation

• Answer Diagnosis Map
• Natural Language Processing
• Targeted Feedback