ICLR Points: How Many ICLR Publications Is One Paper in Each Area?

2025-03-20

AI Summary

To address the lack of comparable, empirically grounded metrics for assessing scholarly effort across subfields of computer science, this paper introduces “ICLR Points”—the first standardized, evidence-based metric for quantifying academic effort. Leveraging DBLP data on top-tier conference publications (ICLR, ICML, NeurIPS) from 2019–2023 and CSRankings faculty data, we employ statistical modeling and cross-domain normalization to estimate the average effort required to publish one paper in each of 27 subfields. Results show that systems-oriented subfields demand significantly higher effort per publication than AI-focused ones—aligning with expert consensus and validating the metric’s construct validity. ICLR Points have been deployed in cross-disciplinary faculty evaluation, enhancing fairness, interpretability, and practical applicability of academic assessment.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Machine Learning: Evaluation and Analysis
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Security and Privacy: Large-scale security measurements

Abstract

Scientific publications significantly impact academic-related decisions in computer science, where top-tier conferences are particularly influential. However, efforts required to produce a publication differ drastically across various subfields. While existing citation-based studies compare venues within areas, cross-area comparisons remain challenging due to differing publication volumes and citation practices. To address this gap, we introduce the concept of ICLR points, defined as the average effort required to produce one publication at top-tier machine learning conferences such as ICLR, ICML, and NeurIPS. Leveraging comprehensive publication data from DBLP (2019--2023) and faculty information from CSRankings, we quantitatively measure and compare the average publication effort across 27 computer science sub-areas. Our analysis reveals significant differences in average publication effort, validating anecdotal perceptions: systems conferences generally require more effort per publication than AI conferences. We further demonstrate the utility of the ICLR points metric by evaluating publication records of current faculties and recent faculty candidates. Our findings highlight how using this metric enables more meaningful cross-area comparisons in academic evaluation processes. Lastly, we discuss the metric's limitations and caution against its misuse, emphasizing the necessity of holistic assessment criteria beyond publication metrics alone.

Problem

• Measure effort differences in publishing across computer science subfields
• Introduce ICLR points for cross-area conference publication comparisons
• Evaluate faculty publication records using standardized effort metrics

Innovation

• Introducing ICLR points for cross-area effort comparison
• Using DBLP and CSRankings data for quantitative analysis
• Evaluating faculty records with ICLR points metric