LLM-as-a-Grader: Practical Insights from Large Language Model for Short-Answer and Report Evaluation

2025-11-13

AI Summary

This study investigates the feasibility and inter-rater agreement of GPT-4o for automated scoring of short-answer quizzes and project reports in an undergraduate computational linguistics course. Method: Conducting the first systematic evaluation of LLM-based grading in a real classroom setting, we quantify alignment between model and teaching assistant (TA) scores using Pearson correlation coefficients and exact match rates. Contribution/Results: GPT-4o achieves a maximum Pearson correlation of 0.98 with human graders; 55% of short-answer responses receive identical scores. Project report evaluations also demonstrate high holistic agreement. Crucially, the study characterizes the stability and variability of LLM scoring behavior on open-ended educational tasks—revealing both robust consistency and context-sensitive divergence. To support reproducibility and methodological advancement in educational automation, we publicly release all source code and annotated sample data, establishing an empirically grounded benchmark for AI-assisted assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLMs) are increasingly explored for educational tasks such as grading, yet their alignment with human evaluation in real classrooms remains underexamined. In this study, we investigate the feasibility of using an LLM (GPT-4o) to evaluate short-answer quizzes and project reports in an undergraduate Computational Linguistics course. We collect responses from approximately 50 students across five quizzes and receive project reports from 14 teams. LLM-generated scores are compared against human evaluations conducted independently by the course teaching assistants (TAs). Our results show that GPT-4o achieves strong correlation with human graders (up to 0.98) and exact score agreement in 55% of quiz cases. For project reports, it also shows strong overall alignment with human grading, while exhibiting some variability in scoring technical, open-ended responses. We release all code and sample data to support further research on LLMs in educational assessment. This work highlights both the potential and limitations of LLM-based grading systems and contributes to advancing automated grading in real-world academic settings.

Problem

• Investigating LLM feasibility for grading student quizzes and reports
• Comparing GPT-4o evaluation accuracy against human teaching assistants
• Assessing alignment variability in technical open-ended response scoring

Innovation

• Using GPT-4o for automated grading tasks
• Comparing LLM scores with human evaluations
• Achieving strong correlation in quiz assessments