BOOST: Bayesian Optimization with Optimal Kernel and Acquisition Function Selection Technique

2025-08-04

AI Summary

Bayesian optimization (BO) performance critically depends on the joint selection of kernel and acquisition functions; however, existing work predominantly optimizes these components in isolation, lacking systematic, automated methods for their coordinated configuration. This paper introduces the first end-to-end framework for joint kernel–acquisition optimization. It employs an offline, lightweight evaluation mechanism that adaptively selects the optimal kernel–acquisition pair from historical BO trajectories, using a reference-set–query-set partitioning strategy. The method requires no online hyperparameter tuning, substantially reducing configuration overhead. Evaluated on synthetic benchmarks and real-world hyperparameter optimization tasks, it achieves an average 23.6% improvement in convergence speed over fixed-configuration BO, while demonstrating superior robustness—particularly under low-budget settings.

Technology Category

• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Machine Learning: Kernel Methods

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The performance of Bayesian optimization (BO), a highly sample-efficient method for expensive black-box problems, is critically governed by the selection of its hyperparameters, including the kernel and acquisition functions. This presents a challenge: an inappropriate combination of these can lead to poor performance and wasted evaluations. While individual improvements to kernel functions (e.g., tree-based kernels, deep kernel learning) and acquisition functions (e.g., multi-step lookahead, tree-based planning) have been explored, the joint and autonomous selection of the best pair of these fundamental hyperparameters has been overlooked. This forces practitioners to rely on heuristics or costly manual training. We propose a simple yet effective framework, BOOST (Bayesian Optimization with Optimal Kernel and Acquisition Function Selection Technique), that automates this selection. BOOST utilizes a lightweight, offline evaluation stage to predict the performance of various kernel-acquisition function pairs and identify the most suitable configuration before expensive evaluations. BOOST partitions data-in-hand into two subsets: a reference subset and a query subset, and it prepares all possible kernel-acquisition pairs from the user's chosen candidates. For each configuration, BOOST conducts internal BO runs using the reference subset, evaluating how effectively each pair guides the search toward the optimum in the unknown query subset, thereby identifying the configuration with the best retrospective performance for future optimization. Experiments on both synthetic benchmark functions and real-world hyperparameter optimization tasks demonstrate that BOOST consistently outperforms standard BO approaches with fixed hyperparameters, highlighting its effectiveness and robustness in diverse problem landscapes.

Problem

• Automates selection of optimal kernel and acquisition functions in Bayesian optimization
• Addresses poor performance from inappropriate kernel-acquisition function combinations
• Eliminates reliance on heuristics or manual tuning for hyperparameter selection

Innovation

• Automates kernel and acquisition function selection
• Uses offline evaluation for performance prediction
• Partitions data for optimal configuration identification