"I understand why I got this grade": Automatic Short Answer Grading with Feedback

2024-06-30

arXiv.org

AI Summary

A high-quality, publicly available dataset supporting automated short-answer grading and explainable feedback generation remains lacking in educational research. To address this, we introduce EngSAF—the first engineering-domain short-answer feedback benchmark (5.8K samples)—covering diverse disciplines and question types from authentic instructional settings. We propose Label-Aware Synthetic Feedback Generation (LASFG), a novel framework integrating grading label guidance with large language model (LLM) zero-shot reasoning and fine-tuning. Feedback quality is ensured through multi-stage human verification and domain-expert annotation. Deployed in the IIT Bombay final examinations, LASFG achieves 92% grading accuracy and produces feedback rated by instructors as comparable to human-generated output in terms of reasonableness and comprehensibility. This work establishes the first engineering-specific ASAG benchmark featuring high-fidelity, expert-verified feedback, advancing automated educational assessment toward explainability, trustworthiness, and practical applicability.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Knowledge Representation and Reasoning: Knowledge Engineering
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The demand for efficient and accurate assessment methods has intensified as education systems transition to digital platforms. Providing feedback is essential in educational settings and goes beyond simply conveying marks as it justifies the assigned marks. In this context, we present a significant advancement in automated grading by introducing Engineering Short Answer Feedback (EngSAF) -- a dataset of 5.8k student answers accompanied by reference answers and questions for the Automatic Short Answer Grading (ASAG) task. The EngSAF dataset is meticulously curated to cover a diverse range of subjects, questions, and answer patterns from multiple engineering domains. We leverage state-of-the-art large language models' (LLMs) generative capabilities with our Label-Aware Synthetic Feedback Generation (LASFG) strategy to include feedback in our dataset. This paper underscores the importance of enhanced feedback in practical educational settings, outlines dataset annotation and feedback generation processes, conducts a thorough EngSAF analysis, and provides different LLMs-based zero-shot and finetuned baselines for future comparison. Additionally, we demonstrate the efficiency and effectiveness of the ASAG system through its deployment in a real-world end-semester exam at the Indian Institute of Technology Bombay (IITB), showcasing its practical viability and potential for broader implementation in educational institutions.

Problem

• Automating short answer grading and feedback using AI
• Lack of public datasets for grading with feedback
• Generating meaningful feedback with large language models

Innovation

• Uses AI for automatic short answer grading
• Introduces EngSAF dataset for feedback generation
• Leverages LLMs with LASFG strategy