Problem
Research questions and friction points this paper is trying to address.
reviewer matching
paper-to-paper matching
multi-dimensional expertise
structured profiling
academic peer review
Beyond Paper-to-Paper: Structured Profiling and Rubric Scoring for Paper-Reviewer Matching
🤖 AI Summary
This work addresses the limitation of existing paper-reviewer matching methods, which predominantly rely on implicit similarity and fail to comprehensively capture reviewers’ multidimensional expertise. The authors propose P2R, a novel framework that introduces explicit, structured expert profiling and a large language model (LLM)-based committee mechanism guided by scoring rules. Specifically, general-purpose LLMs are employed to extract structured profiles—spanning dimensions such as topic, methodology, and application—for both papers and reviewers. Matching proceeds in two stages: an initial coarse retrieval combines semantic and dimensional signals to identify candidate reviewers, followed by fine-grained evaluation via a multi-agent LLM committee that applies predefined rules for multidimensional assessment. Requiring no training, P2R significantly outperforms state-of-the-art methods on NeurIPS, SIGIR, and SciRepEval benchmarks, with ablation studies confirming the contribution of each component.
📝 Abstract
As conference submission volumes continue to grow, accurately recommending suitable reviewers has become a challenge. Most existing methods follow a ``Paper-to-Paper'' matching paradigm, implicitly representing a reviewer by their publication history. However, effective reviewer matching requires capturing multi-dimensional expertise, and textual similarity to past papers alone is often insufficient. To address this gap, we propose P2R, a training-free framework that shifts from implicit paper-to-paper matching to explicit profile-based matching. P2R uses general-purpose LLMs to construct structured profiles for both submissions and reviewers, disentangling them into Topics, Methodologies, and Applications. Building on these profiles, P2R adopts a coarse-to-fine pipeline to balance efficiency and depth. It first performs hybrid retrieval that combines semantic and aspect-level signals to form a high-recall candidate pool, and then applies an LLM-based committee to evaluate candidates under strict rubrics, integrating both multi-dimensional expert views and a holistic Area Chair perspective. Experiments on NeurIPS, SIGIR, and SciRepEval show that P2R consistently outperforms state-of-the-art baselines. Ablation studies further verify the necessity of each component. Overall, P2R highlights the value of explicit, structured expertise modeling and offers practical guidance for applying LLMs to reviewer matching.
Innovation
Methods, ideas, or system contributions that make the work stand out.
structured profiling
rubric scoring
paper-reviewer matching
large language models
coarse-to-fine retrieval
🔎 Similar Papers
No similar papers found.
