OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews

2024-12-16

arXiv.org

AI Summary

AI/ML researchers lack high-quality pre-submission review tools to assess manuscript quality before formal peer review. Method: We propose OpenReviewer—the first large language model system deeply specialized for peer review—built upon the Llama-3 architecture, equipped with precise PDF parsing (supporting mathematical formulas and tables), supervised fine-tuned on 79,000 real expert reviews, and employing a template-driven structured generation mechanism. Contribution/Results: On a test set of 400 papers, OpenReviewer achieves significantly higher criticality in reviews than GPT-4 and Claude-3.5 (p < 0.01, two-sample Kolmogorov–Smirnov test), and its recommendation score distribution closely matches human reviewers’. OpenReviewer fills a critical gap in automated academic pre-review tools, delivering trustworthy, conference-compliant feedback to authors for iterative manuscript improvement—complementing, not replacing, human review.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We present OpenReviewer, an open-source system for generating high-quality peer reviews of machine learning and AI conference papers. At its core is Llama-OpenReviewer-8B, an 8B parameter language model specifically fine-tuned on 79,000 expert reviews from top ML conferences. Given a PDF paper submission and review template as input, OpenReviewer extracts the full text, including technical content like equations and tables, and generates a structured review following conference-specific guidelines. Our evaluation on 400 test papers shows that OpenReviewer produces significantly more critical and realistic reviews compared to general-purpose LLMs like GPT-4 and Claude-3.5. While other LLMs tend toward overly positive assessments, OpenReviewer's recommendations closely match the distribution of human reviewer ratings. The system provides authors with rapid, constructive feedback to improve their manuscripts before submission, though it is not intended to replace human peer review. OpenReviewer is available as an online demo and open-source tool.

Problem

• Generates critical peer reviews for AI and ML papers.
• Fine-tuned on 79,000 expert reviews for accuracy.
• Provides rapid, constructive feedback to improve manuscripts.

Innovation

• Fine-tuned 8B parameter model on expert reviews
• Extracts and reviews full text from PDF submissions
• Generates structured reviews matching human ratings