Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism

2026-01-21

AI Summary

This work addresses the challenge of inconsistent and low-quality peer reviews that often undermine best-paper selection at large-scale machine learning and artificial intelligence conferences. The authors propose an author-assisted mechanism that leverages isotonic regression to elicit truthful rankings of submitted papers from authors themselves, which are then used to calibrate original review scores for more accurate estimation of true paper quality. The mechanism guarantees incentive compatibility under the mild assumption that authors’ utility functions are non-decreasing and additive—a significant relaxation of the strong convexity assumptions required by conventional approaches. Empirical evaluation on real review data from ICLR (2019–2023) and NeurIPS (2021–2023) demonstrates that the proposed method substantially improves the accuracy of best-paper award decisions.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Multiagent Systems: Mechanism Design
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Machine learning and artificial intelligence conferences such as NeurIPS and ICML now regularly receive tens of thousands of submissions, posing significant challenges to maintaining the quality and consistency of the peer review process. This challenge is particularly acute for best paper awards, which are an important part of the peer review process, yet whose selection has increasingly become a subject of debate in recent years. In this paper, we introduce an author-assisted mechanism to facilitate the selection of best paper awards. Our method employs the Isotonic Mechanism for eliciting authors'assessments of their own submissions in the form of a ranking, which is subsequently utilized to adjust the raw review scores for optimal estimation of the submissions'ground-truth quality. We demonstrate that authors are incentivized to report truthfully when their utility is a convex additive function of the adjusted scores, and we validate this convexity assumption for best paper awards using publicly accessible review data of ICLR from 2019 to 2023 and NeurIPS from 2021 to 2023. Crucially, in the special case where an author has a single quota -- that is, may nominate only one paper -- we prove that truthfulness holds even when the utility function is merely nondecreasing and additive. This finding represents a substantial relaxation of the assumptions required in prior work. For practical implementation, we extend our mechanism to accommodate the common scenario of overlapping authorship. Finally, simulation results demonstrate that our mechanism significantly improves the quality of papers selected for awards.

Problem

• best paper awards
• peer review
• machine learning conferences
• review quality
• AI conference

Innovation

• Isotonic Mechanism
• best paper award
• truthful elicitation
• peer review
• author-assisted evaluation