Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track

2025-06-23

AI Summary

Leading machine learning conferences lack systematic mechanisms to correct erroneous, misleading, or incomplete conclusions in published papers. Method: We propose a dedicated “Refutations and Critiques” (R&C) track—the first formal, peer-reviewed, and archivable academic correction channel for ML conferences. Our design integrates scholarly commentary norms, reforms to peer review, and empirical case studies, yielding a complete R&C framework specifying submission criteria, double-blind review, categorized publication of outcomes, and bidirectional cross-linking with original papers. Feasibility is validated via integration into ICLR 2025’s oral presentation process. Contribution/Results: The R&C track significantly enhances transparency, reproducibility, and self-correcting capacity in ML research. It establishes a scalable, rigorous, and implementable governance model for the AI community, advancing scientific integrity through structured post-publication scrutiny.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Science progresses by iteratively advancing and correcting humanity's understanding of the world. In machine learning (ML) research, rapid advancements have led to an explosion of publications, but have also led to misleading, incorrect, flawed or perhaps even fraudulent studies being accepted and sometimes highlighted at ML conferences due to the fallibility of peer review. While such mistakes are understandable, ML conferences do not offer robust processes to help the field systematically correct when such errors are made.This position paper argues that ML conferences should establish a dedicated "Refutations and Critiques" (R & C) Track. This R & C Track would provide a high-profile, reputable platform to support vital research that critically challenges prior research, thereby fostering a dynamic self-correcting research ecosystem. We discuss key considerations including track design, review principles, potential pitfalls, and provide an illustrative example submission concerning a recent ICLR 2025 Oral. We conclude that ML conferences should create official, reputable mechanisms to help ML research self-correct.

Problem

• ML conferences lack mechanisms to correct flawed studies
• Need for reputable platform to critique prior research
• Establishing a Refutations and Critiques Track in ML

Innovation

• Establish dedicated Refutations and Critiques Track
• Provide reputable platform for critical research
• Foster self-correcting ML research ecosystem