Disparities in Peer Review Tone and the Role of Reviewer Anonymity

2025-07-19

AI Summary

This study identifies implicit linguistic bias in peer review, demonstrating that review tone—measured via sentiment polarity and supportive language—is systematically influenced by author gender, race, and institutional prestige; reviewer anonymity fails to mitigate—and in some cases exacerbates—negative evaluations of underrepresented groups. Method: Leveraging the first large-scale integration of natural language processing, fine-grained sentiment analysis, and multilevel statistical modeling, we analyze over 80,000 real anonymized and identified review texts. Contribution/Results: We find that disclosing reviewer identity systematically alters linguistic expression and attenuates demographic effects on review tone—directly challenging the long-standing “anonymity ensures fairness” assumption. These findings provide empirical grounding and policy-relevant evidence for redesigning peer review toward greater equity and transparency.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

The peer review process is often regarded as the gatekeeper of scientific integrity, yet increasing evidence suggests that it is not immune to bias. Although structural inequities in peer review have been widely debated, much less attention has been paid to the subtle ways in which language itself may reinforce disparities. This study undertakes one of the most comprehensive linguistic analyses of peer review to date, examining more than 80,000 reviews in two major journals. Using natural language processing and large-scale statistical modeling, it uncovers how review tone, sentiment, and supportive language vary across author demographics, including gender, race, and institutional affiliation. Using a data set that includes both anonymous and signed reviews, this research also reveals how the disclosure of reviewer identity shapes the language of evaluation. The findings not only expose hidden biases in peer feedback, but also challenge conventional assumptions about anonymity's role in fairness. As academic publishing grapples with reform, these insights raise critical questions about how review policies shape career trajectories and scientific progress.

Problem

• Examines bias in peer review tone across author demographics
• Analyzes impact of reviewer anonymity on evaluation language
• Reveals hidden disparities in peer feedback affecting scientific careers

Innovation

• Natural language processing analyzes review tone
• Statistical modeling examines author demographics impact
• Compares anonymous versus signed reviews linguistically