OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models

2025-05-24

AI Summary

The large language model (LLM) era urgently requires high-quality, authentic, and structured scientific reasoning data to advance trustworthy AI research. Method: This paper pioneers the systematic adoption of OpenReview—a dynamic, expert-curated academic interaction dataset encompassing papers, peer reviews, author rebuttals, meta-reviews, and final decisions—as a scarce source of expert-level alignment data. Our approach emphasizes data governance, standardized benchmark design, an ethical usage framework, and community co-governance—model-agnostic by design. Contributions: (1) We establish OpenReview’s irreplaceable value across three dimensions: scalability of review-based evaluation, authenticity of open scientific benchmarks, and empirical rigor in alignment research; (2) we introduce the first standardized OpenReview usage guidelines and a shared-responsibility agreement; and (3) we catalyze three novel research paradigms—review-augmented reasoning, open scientific benchmarking, and value-aligned AI—thereby laying a robust academic infrastructure for explainable, value-consistent LLMs.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In the era of large language models (LLMs), high-quality, domain-rich, and continuously evolving datasets capturing expert-level knowledge, core human values, and reasoning are increasingly valuable. This position paper argues that OpenReview -- the continually evolving repository of research papers, peer reviews, author rebuttals, meta-reviews, and decision outcomes -- should be leveraged more broadly as a core community asset for advancing research in the era of LLMs. We highlight three promising areas in which OpenReview can uniquely contribute: enhancing the quality, scalability, and accountability of peer review processes; enabling meaningful, open-ended benchmarks rooted in genuine expert deliberation; and supporting alignment research through real-world interactions reflecting expert assessment, intentions, and scientific values. To better realize these opportunities, we suggest the community collaboratively explore standardized benchmarks and usage guidelines around OpenReview, inviting broader dialogue on responsible data use, ethical considerations, and collective stewardship.

Problem

• Enhancing quality, scalability, and accountability of peer review
• Creating open-ended benchmarks from expert deliberation
• Supporting alignment research via expert assessments and values

Innovation

• Enhancing peer review quality and scalability
• Creating expert-rooted open-ended benchmarks
• Supporting alignment research via expert interactions