Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs

2025-09-22

AI Summary

Bayesian optimization (BO) performance critically depends on the suitability of the Gaussian process (GP) kernel, yet conventional approaches rely on fixed or heuristic kernels, often resulting in slow convergence or suboptimal solutions. To address this, we propose CAKE, the first framework to integrate large language models (LLMs) into GP kernel design for BO. CAKE employs LLMs as data-driven mutation and crossover operators to generate context-aware candidate kernels. Coupled with the BAKER selection strategy—which jointly evaluates kernels via the Bayesian Information Criterion (BIC) and Expected Improvement (EI)—CAKE enables adaptive kernel selection and evolution. Extensive experiments across hyperparameter tuning, controller parameter optimization, and photonic chip design demonstrate that CAKE significantly improves both convergence speed and solution quality under limited function evaluations. The implementation is publicly available.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The efficiency of Bayesian optimization (BO) relies heavily on the choice of the Gaussian process (GP) kernel, which plays a central role in balancing exploration and exploitation under limited evaluation budgets. Traditional BO methods often rely on fixed or heuristic kernel selection strategies, which can result in slow convergence or suboptimal solutions when the chosen kernel is poorly suited to the underlying objective function. To address this limitation, we propose a freshly-baked Context-Aware Kernel Evolution (CAKE) to enhance BO with large language models (LLMs). Concretely, CAKE leverages LLMs as the crossover and mutation operators to adaptively generate and refine GP kernels based on the observed data throughout the optimization process. To maximize the power of CAKE, we further propose BIC-Acquisition Kernel Ranking (BAKER) to select the most effective kernel through balancing the model fit measured by the Bayesian information criterion (BIC) with the expected improvement at each iteration of BO. Extensive experiments demonstrate that our fresh CAKE-based BO method consistently outperforms established baselines across a range of real-world tasks, including hyperparameter optimization, controller tuning, and photonic chip design. Our code is publicly available at https://github.com/cake4bo/cake.

Problem

• Traditional Bayesian optimization relies on fixed kernel selection strategies
• Poor kernel choices lead to slow convergence and suboptimal solutions
• Adaptive kernel design is needed for better exploration-exploitation balance

Innovation

• LLMs adaptively generate GP kernels
• BIC-Acquisition balances model fit and improvement
• Context-Aware Kernel Evolution enhances Bayesian optimization