OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation

2026-02-09

AI Summary

Existing reviewer recommendation systems are hindered by the scarcity of large-scale real-world data and evaluation protocols that inadequately reflect actual editorial workflows, often suffering from insufficient semantic representation and weak interpretability. To address these limitations, this work introduces OmniReview, a novel dataset comprising 202,756 verified review records, and proposes a three-tiered evaluation framework that more faithfully simulates the peer review process. Furthermore, we present Pro-MMoE, a model that integrates semantic scholar profiles generated by large language models with a task-adaptive multi-gate mixture-of-experts (MMoE) architecture to enable multi-objective optimization and fine-grained expertise matching. Experimental results demonstrate that our approach achieves state-of-the-art performance on six out of seven evaluation metrics, significantly enhancing both recommendation accuracy and practical utility, thereby establishing a new benchmark for reviewer recommendation tasks.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Data Mining & Knowledge Management: Recommender Systems
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Academic peer review remains the cornerstone of scholarly validation, yet the field faces some challenges in data and methods. From the data perspective, existing research is hindered by the scarcity of large-scale, verified benchmarks and oversimplified evaluation metrics that fail to reflect real-world editorial workflows. To bridge this gap, we present OmniReview, a comprehensive dataset constructed by integrating multi-source academic platforms encompassing comprehensive scholarly profiles through the disambiguation pipeline, yielding 202, 756 verified review records. Based on this data, we introduce a three-tier hierarchical evaluaion framework to assess recommendations from recall to precise expert identification. From the method perspective, existing embedding-based approaches suffer from the information bottleneck of semantic compression and limited interpretability. To resolve these method limitations, we propose Profiling Scholars with Multi-gate Mixture-of-Experts (Pro-MMoE), a novel framework that synergizes Large Language Models (LLMs) with Multi-task Learning. Specifically, it utilizes LLM-generated semantic profiles to preserve fine-grained expertise nuances and interpretability, while employing a Task-Adaptive MMoE architecture to dynamically balance conflicting evaluation goals. Comprehensive experiments demonstrate that Pro-MMoE achieves state-of-the-art performance across six of seven metrics, establishing a new benchmark for realistic reviewer recommendation.

Problem

• peer review
• reviewer recommendation
• benchmark dataset
• evaluation framework
• semantic compression

Innovation

• Large Language Models
• Reviewer Recommendation
• Multi-task Learning
• Mixture-of-Experts
• Benchmark Dataset