Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review

2026-05-24

AI Summary

This study addresses the challenges surrounding the effectiveness, fairness, and susceptibility to strategic behavior of author self-assessment mechanisms in peer review at top machine learning conferences. Building on the ICML 2023 review experiment, it formulates the peer review process as a statistical estimation problem and proposes an integrated framework that jointly leverages author self-evaluations, reviewer rankings, and structured metadata. The work introduces a strategyproof mechanism based on isotonic regression, which enhances review quality while preserving fairness. By systematically responding to both theoretical and practical debates about self-assessment, this research lays a methodological foundation and offers a viable pathway toward designing human-centered, trustworthy peer review systems in the era of generative AI.

Abstract

This article is the rejoinder to ``The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review,'' to appear in the Journal of the American Statistical Association with discussion. To address the practical and theoretical points raised by the discussants, we organize our response around four core themes: (i) formulating peer review as a statistical estimation problem; (ii) mitigating equity and strategic concerns in the deployment of the Isotonic Mechanism; (iii) incorporating complementary signals such as reviewer rankings and structured metadata; and (iv) exploring a human-centered framework for peer review in the era of generative AI.

Problem

• peer review
• author self-assessment
• equity
• strategic behavior
• review quality

Innovation

• statistical estimation
• Isotonic Mechanism
• peer review
• equity in AI
• human-centered AI