Best Practices For Empirical Meta-Algorithmic Research Guidelines from the COSEAL Research Network

2025-12-18

AI Summary

Empirical studies of meta-algorithms—such as algorithm selection, configuration, and scheduling—suffer from poor reproducibility and high bias risk due to excessive degrees of freedom in experimental design and fragmented community practices. Method: This paper introduces the first systematic integration of long-standing best practices from the COSEAL community across subfields, yielding a unified, dynamically evolving methodology framework spanning the entire experimental lifecycle: problem formulation → experimental design → execution → analysis → result presentation. Grounded in empirical methodology, rigorous experimental design, statistical standards, and principles of scientific communication, it emphasizes controlled variable management, benchmark standardization, and result transparency. Contribution/Results: The framework significantly reduces experimental bias, enhances cross-study comparability, and strengthens scientific rigor. It has been adopted for onboarding new researchers and informing journal review criteria.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Multiagent Systems: Mechanism Design
• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Empirical research on meta-algorithmics, such as algorithm selection, configuration, and scheduling, often relies on extensive and thus computationally expensive experiments. With the large degree of freedom we have over our experimental setup and design comes a plethora of possible error sources that threaten the scalability and validity of our scientific insights. Best practices for meta-algorithmic research exist, but they are scattered between different publications and fields, and continue to evolve separately from each other. In this report, we collect good practices for empirical meta-algorithmic research across the subfields of the COSEAL community, encompassing the entire experimental cycle: from formulating research questions and selecting an experimental design, to executing ex- periments, and ultimately, analyzing and presenting results impartially. It establishes the current state-of-the-art practices within meta-algorithmic research and serves as a guideline to both new researchers and practitioners in meta-algorithmic fields.

Problem

• Addresses scalability and validity issues in meta-algorithmic experiments
• Consolidates scattered best practices across subfields into unified guidelines
• Provides comprehensive guidance for the entire experimental research cycle

Innovation

• Collects best practices across meta-algorithmic subfields
• Encompasses entire experimental cycle from design to analysis
• Establishes guidelines for scalable and valid research