MERIT: Matching Expertise via Rubric-Informed Training for Reviewer Assignment

2026-05-26

AI Summary

This work addresses the challenge of accurately matching papers to reviewers in large-scale academic peer review, where existing approaches are limited by coarse similarity metrics or non-scalable manual annotations. The authors propose MERIT, a two-stage framework that first leverages large language models (LLMs) to generate reward signals based on fine-grained rubric-guided expertise criteria, then trains a 4B-parameter reviewer evaluator via reinforcement learning. In the second stage, the knowledge of this evaluator is distilled into an efficient embedding-based retriever to enable scalable reviewer assignment. This study is the first to formulate granular expertise matching as a supervised signal, achieving state-of-the-art performance on the LR-Bench and CMU Gold datasets. Notably, the specialized evaluator outperforms larger general-purpose LLMs on reviewer-paper fit classification tasks.

Abstract

Matching submissions with suitable reviewers at scale is a growing challenge for major venues, yet existing approaches either rely on coarse proxy signals that conflate general relatedness with true suitability, or require expensive human annotations that are difficult to scale for training. We propose MERIT, a two-stage framework that bridges this gap by converting criterion-level expertise matching into scalable suitability supervision. In the first stage, we train a reviewer assessor via reinforcement learning to identify the expertise dimensions a paper requires, match them against the reviewer's prior work, and produce a suitability decision, with rewards provided by an LLM judge guided by paper-specific expertise rubrics. In the second stage, we distill the assessor's predictions into an embedding-based retriever for efficient large-scale assignment. Experiments show that our 4B reviewer assessor outperforms larger general-purpose LLMs on suitability classification, and the resulting retriever achieves state-of-the-art performance across LR-Bench and the CMU Gold dataset. Our code is available at https://github.com/Luli3220/MERIT.

Problem

• reviewer assignment
• expertise matching
• large-scale matching
• suitability assessment
• academic peer review

Innovation

• reviewer assignment
• expertise matching
• rubric-informed training
• reinforcement learning
• knowledge distillation