Scaling Success: A Systematic Review of Peer Grading Strategies for Accuracy, Efficiency, and Learning in Contemporary Education

2025-08-08

arXiv.org

AI Summary

Peer assessment offers scalability and pedagogical value in large-scale online learning, yet challenges persist regarding accuracy, fairness, reliability, and student engagement. This study systematically reviews 122 empirical peer assessment studies and develops a multidimensional classification framework encompassing evaluation modalities and reviewer weighting strategies. It introduces, for the first time, formative feedback quality as an explicit factor in summative scoring—proposing a hybrid weighting strategy that jointly incorporates feedback quality and reviewer competence. Leveraging educational data mining and cross-study comparative analysis, the research identifies causal mechanisms through which design choices influence both assessment quality and learning outcomes. Findings yield actionable, evidence-based design guidelines for practitioners and researchers, supporting robust implementation of peer assessment in digital learning environments.

Technology Category

• Multiagent Systems: Mechanism Design
• Machine Learning: Learning Preferences or Rankings
• Game Theory and Economic Paradigms: Mechanism Design

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Peer grading has emerged as a scalable solution for assessment in large and online classrooms, offering both logistical efficiency and pedagogical value. However, designing effective peer-grading systems remains challenging due to persistent concerns around accuracy, fairness, reliability, and student engagement. This paper presents a systematic review of 122 peer-reviewed studies on peer grading spanning over four decades. Drawing from this literature, we propose a comprehensive taxonomy that organizes peer grading systems along two key dimensions: (1) evaluation approaches and (2) reviewer weighting strategies. We analyze how different design choices impact grading accuracy, fairness, student workload, and learning outcomes. Our findings highlight the strengths and limitations of each method. Notably, we found that formative feedback -- often regarded as the most valuable aspect of peer assessment -- is seldom incorporated as a quality-based weighting factor in summative grade synthesis techniques. Furthermore, no single reviewer weighting strategy proves universally optimal; each has its trade-offs. Hybrid strategies that combine multiple techniques could show the greatest promise. Our taxonomy offers a practical framework for educators and researchers aiming to design peer grading systems that are accurate, equitable, and pedagogically meaningful.

Problem

• Designing effective peer grading systems faces accuracy and fairness challenges
• Current systems rarely incorporate formative feedback in grade synthesis
• No single reviewer weighting strategy proves universally optimal for grading

Innovation

• Systematic review of peer grading studies
• Taxonomy with evaluation and weighting dimensions
• Hybrid strategies combining multiple techniques