Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviews

2025-09-16

AI Summary

This study investigates implicit biases in large language models (LLMs) when generating peer review comments—specifically, biases tied to author institutional prestige and gender—and reveals their potential threat to fairness and reliability in scholarly evaluation. Through rigorously controlled comparative experiments and analysis of sensitive metadata (e.g., affiliation rank, gender proxies), we quantify, for the first time, a dual bias: strong preference for authors from top-ranked institutions and a statistically significant, albeit subtle, gender bias. We propose a novel soft-scoring method grounded in token probability distributions, enabling detection and quantification of latent biases. Experimental results demonstrate that this approach effectively amplifies and localizes bias signals, yielding a reproducible benchmark framework for assessing fairness in LLM-assisted peer review and informing targeted mitigation strategies.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The adoption of large language models (LLMs) is transforming the peer review process, from assisting reviewers in writing more detailed evaluations to generating entire reviews automatically. While these capabilities offer exciting opportunities, they also raise critical concerns about fairness and reliability. In this paper, we investigate bias in LLM-generated peer reviews by conducting controlled experiments on sensitive metadata, including author affiliation and gender. Our analysis consistently shows affiliation bias favoring institutions highly ranked on common academic rankings. Additionally, we find some gender preferences, which, even though subtle in magnitude, have the potential to compound over time. Notably, we uncover implicit biases that become more evident with token-based soft ratings.

Problem

• Investigating bias in LLM-generated peer reviews
• Analyzing affiliation and gender-based preferences in evaluations
• Uncovering implicit biases through token-based soft ratings

Innovation

• Detecting bias in LLM peer reviews
• Using controlled experiments on metadata
• Analyzing token-based soft ratings