Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review

2025-01-22

AI Summary

Current peer review suffers from inefficiency, high cost, subjective bias, and insufficient fairness—impeding scientific communication. This paper proposes a data-driven open peer review framework. First, leveraging large-scale open review data, we construct an unbiased reviewer quality estimation model, empirically demonstrating that review competence is uncorrelated with author reputation. Second, we design a consensus consistency metric and a Bayesian weighting mechanism to dynamically calibrate review opinion weights, thereby improving the accuracy of paper quality estimation. Third, we introduce a counter-Matthew-effect incentive mechanism to promote equitable distribution of review effort across submissions. Experiments on open review data from NeurIPS and ICML show that our approach significantly enhances the reliability of paper quality prediction (reducing RMSE by 18.7%), validating the scalability, fairness, and transparency of this new paradigm.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

This study proposes a data-driven framework for enhancing the accuracy and efficiency of scientific peer review through an open, bottom-up process that estimates reviewer quality. Traditional closed peer review systems, while essential for quality control, are often slow, costly, and subject to biases that can impede scientific progress. Here, we introduce a method that evaluates individual reviewer reliability by quantifying agreement with community consensus scores and applying Bayesian weighting to refine paper quality assessments. We analyze open peer review data from two major scientific conferences, and demonstrate that reviewer-specific quality scores significantly improve the reliability of paper quality estimation. Perhaps surprisingly, we find that reviewer quality scores are unrelated to authorship quality. Our model incorporates incentive structures to recognize high-quality reviewers and encourage broader coverage of submitted papers, thereby mitigating the common"rich-get-richer"pitfall of social media. These findings suggest that open peer review, with mechanisms for estimating and incentivizing reviewer quality, offers a scalable and equitable alternative for scientific publishing, with potential to enhance the speed, fairness, and transparency of the peer review process.

Problem

• Scientific Peer Review
• Bias Mitigation
• Efficiency Improvement

Innovation

• Bayesian Weighting
• Reviewer Effectiveness
• Scientific Progress Acceleration