Gender and Discipline Shape Length, Content and Tone of Grant Peer Review Reports

2025-06-30

AI Summary

This study investigates how gender and disciplinary background systematically influence the length, content, and tone of peer review reports for research funding applications—thereby potentially compromising evaluation fairness and consistency. Analyzing nearly 40,000 review reports (comprising over 1.3 million sentences) from a major national funding agency, we employ supervised machine learning to conduct large-scale, multi-dimensional quantitative text analysis—the first such empirical examination of systematic textual biases in grant peer review. Results reveal that reviewers identifying as women produce reports more aligned with formal evaluation criteria and exhibit more positive sentiment; reports in humanities and social sciences are significantly longer and more critically oriented, whereas those in science, technology, engineering, and mathematics emphasize applicants’ track records and are more concise. These findings uncover structural sources of implicit bias in peer review, offering critical empirical evidence and methodological tools to enhance the scientific rigor, equity, and inclusivity of funding decision-making.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Application Domains: Humanities & Computational Social Science
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Sentiment analysis and opinion mining

Abstract

Peer review by experts is central to the evaluation of grant proposals, but little is known about how gender and disciplinary differences shape the content and tone of grant peer review reports. We analyzed 39,280 review reports submitted to the Swiss National Science Foundation between 2016 and 2023, covering 11,385 proposals for project funding across 21 disciplines from the Social Sciences and Humanities (SSH), Life Sciences (LS), and Mathematics, Informatics, Natural Sciences, and Technology (MINT). Using supervised machine learning, we classified over 1.3 million sentences by evaluation criteria and sentiment. Reviews in SSH were significantly longer and more critical, with less focus on the applicant's track record, while those in MINT were more concise and positive, with a higher focus on the track record, as compared to those in LS. Compared to male reviewers, female reviewers write longer reviews that more closely align with the evaluation criteria and express more positive sentiments. Female applicants tend to receive reviews with slightly more positive sentiment than male applicants. Gender and disciplinary culture influence how grant proposals are reviewed - shaping the tone, length, and focus of peer review reports. These differences have important implications for fairness and consistency in research funding.

Problem

• How gender and discipline affect peer review report content
• Differences in review tone and length across disciplines
• Impact of reviewer gender on evaluation criteria alignment

Innovation

• Analyzed 39,280 reviews using supervised machine learning
• Classified 1.3M sentences by criteria and sentiment
• Compared gender and discipline effects on reviews