You can just review things: A digital ethnography of informal peer review

2026-04-17

AI Summary

Traditional peer review is characterized by opacity, loose structure, and publisher dominance, limiting transparency and broad participation. This study systematically defines and empirically investigates informal peer review—a phenomenon integrating three variants of open peer review—for the first time. Employing a cross-platform digital ethnography approach, the research combines participant observation with open- and axial-coding qualitative analysis to trace reviewing practices and metacommentary across 15 online communities. Findings reveal that informal reviewers are highly diverse in identity, self-organize within rudimentary digital spaces, and deploy deep, unconventional strategies. Despite encountering resistance from authors and publishers, these practices demonstrate emergent potential as an evidence infrastructure. Building on these insights, the study proposes a scalable governance framework and pathways for tool optimization.

Technology Category

• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Application Domains: Humanities & Computational Social Science
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Economics, Online Markets and Human Computation: The sharing economy

Abstract

Across scholarly communities, manuscripts face similar evaluative rituals: editors invite experts to privately assess submissions through formal peer reviews. This closed, loosely structured, and publisher-mediated process is now being supplemented by critiques on open, distributed platforms. We call this practice, a blend of three open peer review variants, informal peer review as it is accessible to outsiders, unmediated by publishers, and conducted across public platforms. Informal peer reviewers range from occasional error detectors to experienced sleuths who identify plagiarism, fraud, errors, conflicts of interest, and conceptual flaws. They may interpret methods, clarify jargon, assess value, and connect to related work. Here, we asked four questions: (1) Who are informal peer reviewers? (2) Where do they work? (3) How do they evaluate research? and (4) What are their impacts? To answer these questions, we conducted a cross-platform digital ethnography with participant observation. We traced discourse across communities over four months and revisited cases after nine and twelve months. From 15 communities, we selected 12 case mentions (10 unique cases) and 8 meta-commentaries from 26 reviewers. Using open and axial coding, we generated 1,080 codes and four themes: reviewers are a motley crew, they self-organize across subpar digital spaces, use deep, uncommon strategies, and they face resistance from authors, publishers, and editors. Informal peer review, we concluded, is a fragile, minimally governed patchwork of people, platforms, and practices, as well as an emerging evidence infrastructure that can be scaled up. We advise advocates and tool-builders to evolve informal review tools, communities, training, and governance by connecting to scholars' values, reducing participation friction, and rewarding attempts to extend the scholarly dialogue.

Problem

• informal peer review
• digital ethnography
• open peer review
• scholarly communication
• research evaluation

Innovation

• informal peer review
• digital ethnography
• open scholarship
• evidence infrastructure
• self-organization