Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization

2025-05-27

AI Summary

Existing black-box optimization benchmarking frameworks suffer from dependency conflicts, tight environment coupling, and poor integration scalability, severely compromising experimental reproducibility. To address these challenges, we propose Bencher—a modular benchmarking framework introducing the novel concept of *benchmark isolation*: it decouples benchmark execution from optimization logic via lightweight, version-agnostic RPC abstractions and isolated virtual environments. Bencher supports cross-platform deployment across local machines, Docker containers, and HPC systems (via Singularity). It unifies modeling of heterogeneous search spaces—including continuous, categorical, and binary domains—enabling reproducible benchmark integration across environments, software versions, and application domains. The framework currently supports one-click integration of 80 real-world benchmarks, with a zero-configuration, lightweight client. Empirical evaluation demonstrates substantial improvements in reliability, reproducibility, and engineering compatibility for black-box optimization algorithm assessment.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

We present Bencher, a modular benchmarking framework for black-box optimization that fundamentally decouples benchmark execution from optimization logic. Unlike prior suites that focus on combining many benchmarks in a single project, Bencher introduces a clean abstraction boundary: each benchmark is isolated in its own virtual Python environment and accessed via a unified, version-agnostic remote procedure call (RPC) interface. This design eliminates dependency conflicts and simplifies the integration of diverse, real-world benchmarks, which often have complex and conflicting software requirements. Bencher can be deployed locally or remotely via Docker or on high-performance computing (HPC) clusters via Singularity, providing a containerized, reproducible runtime for any benchmark. Its lightweight client requires minimal setup and supports drop-in evaluation of 80 benchmarks across continuous, categorical, and binary domains.

Problem

• Decouples benchmark execution from optimization logic
• Eliminates dependency conflicts in diverse benchmarks
• Provides containerized reproducible runtime for benchmarks

Innovation

• Modular benchmarking framework decouples execution and logic
• Isolates benchmarks in virtual Python environments via RPC
• Containerized runtime supports Docker and HPC deployment