Benchmarking that Matters: Rethinking Benchmarking for Practical Impact

2025-11-15

AI Summary

Existing mainstream benchmark suites (e.g., BBOB, CEC) lack fidelity to real-world continuous and mixed-integer optimization problems—failing to reflect their structural characteristics, practical constraints, and information limitations—leading to misuse in algorithm competitions, automated algorithm selection, and industrial decision-making. Method: We propose a next-generation, scenario-driven continuous optimization benchmarking framework featuring: (1) a curated benchmark suite grounded in real-world problems; (2) an interpretable high-dimensional problem feature space coupled with an open-source performance database; (3) native support for multi-objective optimization, noisy environments, and algorithm behavioral analysis; and (4) community-driven, dynamic evolution via a collaborative platform. Contribution/Results: The framework bridges the gap between academic evaluation and industrial requirements, significantly enhancing the practicality, interpretability, and reliability of benchmarks for algorithm selection and deployment decisions. It fosters a sustainable, scientifically rigorous, and engineering-ready benchmarking ecosystem.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Benchmarking has driven scientific progress in Evolutionary Computation, yet current practices fall short of real-world needs. Widely used synthetic suites such as BBOB and CEC isolate algorithmic phenomena but poorly reflect the structure, constraints, and information limitations of continuous and mixed-integer optimization problems in practice. This disconnect leads to the misuse of benchmarking suites for competitions, automated algorithm selection, and industrial decision-making, despite these suites being designed for different purposes. We identify key gaps in current benchmarking practices and tooling, including limited availability of real-world-inspired problems, missing high-level features, and challenges in multi-objective and noisy settings. We propose a vision centered on curated real-world-inspired benchmarks, practitioner-accessible feature spaces and community-maintained performance databases. Real progress requires coordinated effort: A living benchmarking ecosystem that evolves with real-world insights and supports both scientific understanding and industrial use.

Problem

• Current benchmarking poorly reflects real-world optimization problem structures and constraints
• There is limited availability of real-world-inspired problems in evolutionary computation
• Benchmarking suites are misused for competitions and industrial decision-making

Innovation

• Curated real-world-inspired benchmarks for optimization
• Practitioner-accessible feature spaces implementation
• Community-maintained performance databases ecosystem