Which Optimizer, At What Budget? A Tournament of Optimizers for Search-Based SE

2026-07-13

AI Summary

This study addresses the absence of a universally optimal optimizer in search-based software engineering and the instability of existing recommendations—such as NSGA-II—across varying evaluation budgets. Conducting large-scale experiments across 106 software engineering tasks, the authors evaluate 20 optimization algorithms under four distinct budget conditions. By integrating task characteristics with budget constraints, they perform clustering and performance analysis to propose a lightweight, budget-aware optimizer selection strategy. Relying solely on two easily obtainable task attributes and the evaluation budget, this approach uses a lookup-table mechanism to predict high-performing optimizers, achieving performance on par with or better than the ex post facto best choice on approximately 75% of held-out tasks. The complete experimental suite is publicly released to support reproducible research.

Abstract

Configuring and tuning modern software is unavoidable, expensive, and error-prone: a single system can expose hundreds of interacting options, and scoring one setting can mean a full build or test run. The standard response is automated optimization, but the number of available optimizers is large and growing. And some of the guidance for selecting among them is misleading: NSGA-II, for example, is widely recommended, yet other algorithms reach the same results using only 1/20th as many evaluations. To help practitioners make better choices about tools to configure their systems, we cluster 20 optimizers, based on six assumptions about the data. Next, we run a tournament across those optimizers, using 106 SE optimization tasks at four labeling budgets (taking 14,000+ CPU hours). We find that no optimizer wins outright. The best one migrates with the budget (from a geometric active learner when labels are scarce to differential evolution when labels are plentiful) so a winner "crowned" at one budget is wrong at another on up to half our tasks. Running such a tournament for every new domain is impractical due to its CPU cost. Fortunately, we find that those 14,000 hours can be replaced by a table lookup over two cheap-to-obtain task attributes (plus the labeling budget). Predictions from this table tie or beat a hindsight oracle on $\approx 75%$ of held-out tasks. To support open science, our tournament and replication package are open-sourced for SBSE researchers and practitioners at https://github.com/KKGanguly/OptimizerTournament.

Problem

• optimizer selection
• search-based software engineering
• labeling budget
• configuration tuning
• empirical comparison

Innovation

• optimizer selection
• search-based software engineering
• labeling budget
• empirical evaluation
• meta-learning