Bayesian Optimization by Kernel Regression and Density-based Exploration

2025-02-10

AI Summary

Bayesian optimization (BO) suffers from high computational overhead—cubic in the number of observations for GP inference and quartic overall (O(T⁴))—when optimizing expensive black-box functions. To address this, we propose BOKE, an efficient BO framework that integrates kernel regression (KR) for surrogate modeling with density-driven exploration. Its core innovation is the first use of KR to replace Gaussian processes, coupled with a novel kernel-density-estimated upper confidence bound (UCB) acquisition function that adaptively balances exploration and exploitation. We theoretically establish BOKE’s global convergence and robustness under mild assumptions. Crucially, BOKE reduces time complexity to O(T²), outperforming standard GP-based BO by one to two orders of magnitude in training speed. Empirical evaluations on synthetic benchmarks and real-world optimization tasks demonstrate that BOKE matches the optimization performance of GP-BO while enabling scalable deployment in resource-constrained engineering applications.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Kernel Methods

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Bayesian optimization is highly effective for optimizing expensive-to-evaluate black-box functions, but it faces significant computational challenges due to the high computational complexity of Gaussian processes, which results in a total time complexity that is quartic with respect to the number of iterations. To address this limitation, we propose the Bayesian Optimization by Kernel regression and density-based Exploration (BOKE) algorithm. BOKE uses kernel regression for efficient function approximation, kernel density for exploration, and the improved kernel regression upper confidence bound criteria to guide the optimization process, thus reducing computational costs to quadratic. Our theoretical analysis rigorously establishes the global convergence of BOKE and ensures its robustness. Through extensive numerical experiments on both synthetic and real-world optimization tasks, we demonstrate that BOKE not only performs competitively compared to Gaussian process-based methods but also exhibits superior computational efficiency. These results highlight BOKE's effectiveness in resource-constrained environments, providing a practical approach for optimization problems in engineering applications.

Problem

• Reduces computational complexity in Bayesian optimization.
• Introduces BOKE algorithm for efficient function approximation.
• Ensures global convergence and computational efficiency.

Innovation

• Kernel regression for function approximation
• Kernel density for exploration
• Quadratic computational cost reduction