Decision-Making with Deliberation: Meta-reviewing as a Document-grounded Dialogue

2025-08-07

AI Summary

This study addresses the low efficiency and inconsistent quality of manual meta-review synthesis across multiple reviewer reports. We propose a document-dialogue-based meta-review assistance framework that models meta-reviewing as a structured dialogue process. Our method introduces a self-optimizing synthetic data generation strategy: large language models (LLMs) initially generate dialogue samples, which are then iteratively refined using domain-knowledge-guided self-refinement to enhance professionalism and consistency; a lightweight, task-specific dialogue agent is subsequently trained on this curated data. Experiments demonstrate that the agent significantly outperforms general-purpose LLM assistants under synthetic-data supervision and effectively improves both efficiency and judgment consistency in real conference review settings. Our core contributions are: (1) the first scalable, high-quality paradigm for generating meta-review dialogue data, and (2) empirical validation of customized dialogue agents as practical tools for academic peer review support.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Meta-reviewing is a pivotal stage in the peer-review process, serving as the final step in determining whether a paper is recommended for acceptance. Prior research on meta-reviewing has treated this as a summarization problem over review reports. However, complementary to this perspective, meta-reviewing is a decision-making process that requires weighing reviewer arguments and placing them within a broader context. Prior research has demonstrated that decision-makers can be effectively assisted in such scenarios via dialogue agents. In line with this framing, we explore the practical challenges for realizing dialog agents that can effectively assist meta-reviewers. Concretely, we first address the issue of data scarcity for training dialogue agents by generating synthetic data using Large Language Models (LLMs) based on a self-refinement strategy to improve the relevance of these dialogues to expert domains. Our experiments demonstrate that this method produces higher-quality synthetic data and can serve as a valuable resource towards training meta-reviewing assistants. Subsequently, we utilize this data to train dialogue agents tailored for meta-reviewing and find that these agents outperform emph{off-the-shelf} LLM-based assistants for this task. Finally, we apply our agents in real-world meta-reviewing scenarios and confirm their effectiveness in enhancing the efficiency of meta-reviewing.footnote{Code and Data: https://github.com/UKPLab/arxiv2025-meta-review-as-dialog

Problem

• Addressing data scarcity for training meta-review dialogue agents
• Improving synthetic dialogue relevance using LLM self-refinement
• Enhancing meta-review efficiency with tailored dialogue assistants

Innovation

• Generating synthetic data using LLMs
• Training dialogue agents for meta-reviewing
• Enhancing efficiency in real-world scenarios