Problem
Research questions and friction points this paper is trying to address.
research assessment
evaluator noise
post-publication peer review
rating variance
scientific evaluation
Judges matter more than papers in post-publication research assessment
🤖 AI Summary
This study addresses the substantial impact of reviewer subjectivity on the reliability of scholarly evaluations. Leveraging 239,521 peer review records from the H1 Connect platform, the authors employ multilevel linear modeling and variance decomposition to systematically quantify the dominant contribution of reviewer-related variability for the first time. The results reveal that reviewer-level effects account for 61% of the total variance in scores—far exceeding the combined influence of manuscript and journal factors (7%) and demographic or institutional biases such as author gender or affiliation (<1%). These findings demonstrate that “reviewer noise” constitutes the primary source of evaluation bias, prompting the authors to advocate for the implementation of “noise audits” in high-stakes academic assessments to enhance fairness and scientific rigor.
📝 Abstract
Research assessment relies on expert evaluations, yet human judgement is noisy, and it is unclear whether differences in assessment arise primarily from differences in genuine research quality or from unwanted differences between evaluators. While numerous studies highlight disagreement and biases in research assessment, they have not quantified judge-related noise relative to variation in the evaluated works. Here we show, in a large post-publication peer review database, that research assessment is driven more by differences between evaluators than by difference in the evaluated research. We partition variance in 239,521 research quality ratings assigned by 12,649 judges to 193,128 papers from the H1 Connect post-publication peer review platform. Using multilevel models, we decomposed judge-related variation into differences in overall severity and differences in the weighting of scientific attributes. We found that judge-related effects accounted for substantially more variance in ratings than the evaluated papers. In our most detailed model, judge-level effects and judge-specific slopes explained 61% of the total variance, whereas combined paper and journal-level effects accounted for only 7%. By contrast, examined measures of directional bias, such as author gender and global affiliation, explained less than 1% of the variance. We conclude that assessment outcomes were shaped more by the judges than by the papers themselves. Our results demonstrate the necessity of noise audits in high-stakes scientific evaluation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
variance partitioning
multilevel modeling
post-publication peer review
evaluation noise
judge bias
🔎 Similar Papers
No similar papers found.
