From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation

2026-03-18

AI Summary

This work addresses the limitations of current large language models in peer review, which rely on isolated absolute scoring and struggle to generalize across conferences, time periods, and evolving review criteria, often learning narrow scoring heuristics. To overcome this, the authors propose the Comparison-Native Paper Evaluation (CNPE) framework—the first approach to integrate comparative learning throughout data construction, training, and inference. CNPE employs graph-based sampling to select highly discriminative paper pairs and combines supervised fine-tuning with comparison-based reward modeling in reinforcement learning, thereby shifting the paradigm from absolute scoring to global relative ranking. Evaluated across multiple datasets, CNPE achieves an average relative performance gain of 21.8% over the DeepReview-14B baseline and demonstrates strong generalization on five unseen datasets.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently. However, since score scales vary across conferences, time periods, and evaluation criteria, models trained on absolute scores are prone to fitting narrow, context-specific rules rather than developing robust scholarly judgment. To overcome this limitation, we propose shifting paper evaluation from isolated scoring to collaborative ranking. In particular, we design \textbf{C}omparison-\textbf{N}ative framework for \textbf{P}aper \textbf{E}valuation (\textbf{CNPE}), integrating comparison into both data construction and model learning. We first propose a graph-based similarity ranking algorithm to facilitate the sampling of more informative and discriminative paper pairs from a collection. We then enhance relative quality judgment through supervised fine-tuning and reinforcement learning with comparison-based rewards. At inference, the model performs pairwise comparisons over sampled paper pairs and aggregates these preference signals into a global relative quality ranking. Experimental results demonstrate that our framework achieves an average relative improvement of \textbf{21.8\%} over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen datasets. \href{https://github.com/ECNU-Text-Computing/ComparisonReview}{Code}.

Problem

• LLM-based paper evaluation
• absolute scoring
• score scale inconsistency
• scholarly judgment
• scientific paper evaluation

Innovation

• comparison-native framework
• collaborative ranking
• pairwise comparison
• relative quality judgment
• graph-based sampling