FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes

2026-06-18

AI Summary

This study addresses critical limitations in existing AI-based peer review systems, which are predominantly confined to computer science, ignore multi-round review dialogues, and prioritize stylistic imitation over genuine editorial judgment. Leveraging Nature Communications’ mandatory transparent review policy, the authors construct a multi-round peer review dialogue dataset spanning five scientific disciplines. They propose a novel “response-only loss masking” mechanism, built upon the Qwen2.5-7B-Instruct model with LoRA fine-tuning, enabling three core tasks: review generation, review updating, and revision cycle prediction. The method achieves 80.5% accuracy and 78.2% F1-macro in predicting editorial decisions—significantly outperforming baselines such as Gemini—and generates reviews averaging 1,187 words with a ROUGE-L score of 0.154, demonstrating both length and quality closely aligned with human-written reviews.

Abstract

AI systems for peer review fail on three fronts: they train on Computer Science and Machine Learning venues alone, ignore the iterative dialogue that validates science, and evaluate on stylistic mimicry rather than real editorial judgment. We introduce FirstPass, a dataset and fine-tuned model that addresses all three. Curating 3,668 complete multi-round peer-review dialogues from Nature Communications across five scientific domains (biology, chemistry, neuroscience, physics, and earth science), we exploit mandatory transparent peer review (instituted November 2022) and verify 100% content integrity by automated audit. We fine-tune Qwen2.5-7B-Instruct via Low-Rank Adaptation (LoRA) on three tasks: review generation, reviewer updating, and revision-cycle prediction. Our key finding is that response-only loss masking is a prerequisite, not an optimization: without it, accuracy is 62.0%, below the majority baseline; with it, FirstPass achieves 80.5% accuracy and F1-macro 78.2% on predicting editorial outcomes (Standard vs. Extended revision cycles), outperforming Gemini-3.1-flash-lite-preview zero-shot by 10.4 percentage points and all baselines with statistical significance (McNemar p < 0.001). On generation, FirstPass produces reviews averaging 1,187 words, substantially closer to human references (2,155 words) than any baseline, achieving ROUGE-L 0.154 with significant gains over Qwen and DeepSeek zero-shot (p < 0.001). Deployed in the pre-submission loop as an anticipatory scientific co-author, FirstPass simulates expert critique and predicts revision cycle outcomes before submission, giving authors the judgment a trusted colleague would provide, with consistent cross-domain performance across five disciplines.

Problem

• peer review
• editorial judgment
• multi-round dialogue
• scientific validation
• AI evaluation

Innovation

• multi-round peer review
• editorial outcome prediction
• response-only loss masking
• LoRA fine-tuning
• transparent peer review