🤖 AI Summary
The Tree-structured Parzen Estimator (TPE), a mainstream Bayesian optimization method, lacks a systematic understanding of the functional roles, impact patterns, and synergistic interactions among its key hyperparameters (e.g., γ, n_startup, n_ei_candidates).
Method: We conduct comprehensive ablation studies across diverse benchmark functions to empirically dissect these parameters’ behaviors and dependencies. Based on rigorous empirical analysis, we derive an interpretable and robust default configuration strategy.
Contribution/Results: Our recommended configuration significantly outperforms the standard TPE implementation and leading baselines—including SMAC and Hyperopt—across heterogeneous benchmarks, improving both optimization efficiency and stability. All experiments are fully reproducible, with source code publicly released.
📝 Abstract
Recent advances in many domains require more and more complicated experiment design. Such complicated experiments often have many parameters, which necessitate parameter tuning. Tree-structured Parzen estimator (TPE), a Bayesian optimization method, is widely used in recent parameter tuning frameworks. Despite its popularity, the roles of each control parameter and the algorithm intuition have not been discussed so far. In this tutorial, we will identify the roles of each control parameter and their impacts on hyperparameter optimization using a diverse set of benchmarks. We compare our recommended setting drawn from the ablation study with baseline methods and demonstrate that our recommended setting improves the performance of TPE. Our TPE implementation is available at https://github.com/nabenabe0928/tpe/tree/single-opt.