Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization

2026-05-18

AI Summary

This work addresses the challenge of kernel design in high-dimensional Bayesian optimization, where Gaussian process kernels are typically handcrafted and existing automated discovery methods suffer from limited search spaces or reliance on raw observational data, hindering scalability. To overcome these limitations, the authors propose Kernel Discovery, a framework that leverages large language model (LLM)-driven evolutionary strategies to explore a broader space of kernel functions beyond conventional additive and multiplicative compositions—without requiring any observed data. The approach employs a two-stage LLM-based generation process to produce diverse yet functionally valid kernel candidates and incorporates a leave-one-out continuous ranked probability score (LOO-CRPS) criterion to mitigate overfitting. Evaluated on five high-dimensional Bayesian optimization benchmarks, the method achieves an average rank of 1.2 out of 17, significantly outperforming current baselines.

Abstract

Gaussian Process (GP) kernels are central to Bayesian optimization (BO), yet designing effective kernels for high-dimensional problems still relies on extensive manual engineering. Existing automated approaches struggle in high dimensions for two bottlenecks: their kernel search space is limited to additions and multiplications of base kernels, and LLM-based approaches require conditioning on raw observations, which becomes infeasible due to context-length limits and the difficulty of extracting meaningful patterns. We introduce \textbf{Kernel Discovery}, a LLM-driven evolutionary framework for high-dimensional BO that searches a broader kernel space beyond predefined composition rules and does not require conditioning on observations. Motivated by the observation that directly prompting an LLM to generate kernel code yields syntactically varied but functionally identical kernels, we adopt a two-stage approach: an LLM first proposes novel mathematical forms, then a second LLM call converts each form into validated, executable code. We also propose a leave-one-out continuous ranked probability score (LOO-CRPS) as a selection criterion that penalizes overfitted kernels. On five high-dimensional BO benchmarks, our method achieves an average rank of \textbf{1.2 out of 17}, outperforming competitive baselines. We further analyze the discovered kernels to identify which kernels lead to improvements in high-dimensional BO.

Problem

• Bayesian Optimization
• Gaussian Process Kernels
• High-dimensional Optimization
• Automated Kernel Discovery
• Large Language Models

Innovation

• Kernel Discovery
• Bayesian Optimization
• Large Language Models
• High-dimensional Optimization
• Gaussian Process Kernels