ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization

2026-05-07

AI Summary

This work addresses the instability in acquisition function estimation caused by Monte Carlo sampling noise in Bayesian optimization, which often leads to erroneous candidate rankings and suboptimal decisions. To mitigate this issue, the paper proposes the first variance reduction method specifically designed for acquisition estimation noise, based on orthogonalization. The approach employs an orthogonalized acquisition estimator combined with score-function control variates to substantially reduce estimation variance while preserving unbiasedness, thereby significantly improving ranking stability. Furthermore, it integrates a surrogate model with an outer log-transformation to enhance overall robustness. Experimental results demonstrate that the proposed method effectively reduces estimation variance, stabilizes candidate rankings, and achieves superior optimization performance on both numerical benchmarks and neural network hyperparameter tuning tasks.

Abstract

Bayesian optimization is widely used for hyperparameter optimization when model evaluations are expensive; however, noisy acquisition estimates can lead to unstable decisions. We identify acquisition estimation noise as a failure mode that was previously overlooked: even when the surrogate model and acquisition target are correctly specified, finite-sample Monte Carlo error can perturb acquisition values. This can, in turn, flip candidate rankings and lead to suboptimal BO decisions. As a remedy, we aim at variance reduction and propose an orthogonal acquisition estimator that subtracts an optimally weighted score-function control variate, which yields an acquisition residual orthogonal to posterior score directions and which thus reduces Monte Carlo variance. We further introduce OrthoBO: a Bayesian optimization framework that combines our orthogonal acquisition estimator with ensemble surrogates and an outer log transformation. We show theoretically that our estimator preserves the target, leads to variance reduction, and improves pairwise ranking stability. We further verify the theoretical properties of OrthoBO through numerical experiments where our framework reduces acquisition estimation variance, stabilizes candidate rankings, and achieves strong performance. We also demonstrate the downstream utility of OrthoBO in hyperparameter optimization for neural network training and fine-tuning.

Problem

• Bayesian optimization
• hyperparameter optimization
• acquisition estimation noise
• Monte Carlo variance
• ranking instability

Innovation

• orthogonal acquisition estimator
• variance reduction
• Bayesian optimization
• control variate
• hyperparameter optimization