Wasserstein Barycenter Gaussian Process based Bayesian Optimization

2025-05-18

AI Summary

Gaussian process (GP) hyperparameters in Bayesian optimization exhibit high sensitivity to model robustness; maximum likelihood estimation (MLE) tuning is unstable and theoretically challenging. Method: We propose a hyperparameter-free GP ensemble approach: a predefined set of representative hyperparameters is used to construct multiple GP models in parallel, and their posterior distributions are geometrically fused via the Wasserstein barycenter—introduced here for the first time in this context—to yield a single, robust surrogate model. This eliminates online hyperparameter optimization and inherently ensures stability under noise and non-convexity. Contribution/Results: On classical hard-to-optimize benchmarks, our framework significantly outperforms standard Bayesian optimization, achieving more stable convergence and higher success rates in locating global optima. Empirical results validate its effectiveness and robustness under strongly non-convex and high-noise conditions.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Ensemble Methods
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Gaussian Process based Bayesian Optimization is a widely applied algorithm to learn and optimize under uncertainty, well-known for its sample efficiency. However, recently -- and more frequently -- research studies have empirically demonstrated that the Gaussian Process fitting procedure at its core could be its most relevant weakness. Fitting a Gaussian Process means tuning its kernel's hyperparameters to a set of observations, but the common Maximum Likelihood Estimation technique, usually appropriate for learning tasks, has shown different criticalities in Bayesian Optimization, making theoretical analysis of this algorithm an open challenge. Exploiting the analogy between Gaussian Processes and Gaussian Distributions, we present a new approach which uses a prefixed set of hyperparameters values to fit as many Gaussian Processes and then combines them into a unique model as a Wasserstein Barycenter of Gaussian Processes. We considered both"easy"test problems and others known to undermine the extit{vanilla} Bayesian Optimization algorithm. The new method, namely Wasserstein Barycenter Gausssian Process based Bayesian Optimization (WBGP-BO), resulted promising and able to converge to the optimum, contrary to vanilla Bayesian Optimization, also on the most"tricky"test problems.

Problem

• Addresses weaknesses in Gaussian Process fitting for Bayesian Optimization
• Proposes Wasserstein Barycenter to combine multiple Gaussian Processes
• Improves optimization convergence on challenging test problems

Innovation

• Uses prefixed hyperparameters for Gaussian Processes
• Combines models via Wasserstein Barycenter
• Improves convergence in tricky optimization problems