Large Language Models for Departmental Expert Review Quality Scores

2026-01-26

AI Summary

This study presents the first empirical investigation into the applicability of large language models (LLMs) in internal academic peer review within university departments, specifically evaluating their ability to predict expert-assigned quality scores for published papers and comparing their generated review comments with those of human reviewers. Using a dataset of 58 real papers and corresponding internal review records, experiments were conducted with state-of-the-art models including ChatGPT-4o, ChatGPT-4o mini, and Gemini 2.0 Flash. Results reveal a moderate positive correlation between LLM-generated and human expert scores; however, the model-produced reviews tend to be generic, repetitive, and lacking in domain-specific depth or substantive scholarly insight. This suggests that current LLMs rely primarily on superficial cues from titles and abstracts rather than performing genuine academic evaluation, thereby offering empirical evidence of their limitations in authentic peer review contexts.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Presumably, peer reviewers and Large Language Models (LLMs) do very different things when asked to assess research. Still, recent evidence has shown that LLMs have a moderate ability to predict quality scores of published academic journal articles. One untested potential application of LLMs is for internal departmental review, which may be used to support appointment and promotion decisions or to select outputs for national assessments. This study assesses for the first time the extent to which (1) LLM quality scores align with internal departmental quality ratings and (2) LLM reports differ from expert reports. Using a private dataset of 58 published journal articles from the School of Information at the University of Sheffield, together with internal departmental quality ratings and reports, ChatGPT-4o, ChatGPT-4o mini, and Gemini 2.0 Flash scores correlate positively and moderately with internal departmental ratings, whether the input is just title/abstract or the full text. Whilst departmental reviews tended to be more specific and showing field-level knowledge, ChatGPT reports tended to be standardised, more general, repetitive, and with unsolicited suggestions for improvement. The results therefore (a) confirm the ability of LLMs to guess the quality scores of published academic research moderately well, (b) confirm that this ability is a guess rather than an evaluation (because it can be made based on title/abstract alone), (c) extend this ability to internal departmental expert review, and (d) show that LLM reports are less insightful than human expert reports for published academic journal articles.

Problem

• Large Language Models
• peer review
• quality assessment
• academic evaluation
• departmental review

Innovation

• Large Language Models
• expert review
• quality assessment
• academic evaluation
• departmental peer review