Bayesian Optimization under Uncertainty for Training a Scale Parameter in Stochastic Models

2025-10-07

AI Summary

Efficient hyperparameter optimization for scale/precision parameters in stochastic models remains challenging under noisy evaluations. Method: This paper proposes a novel Bayesian optimization framework featuring a statistical surrogate model that enables closed-form analytical expressions of the expected acquisition function. Crucially, it derives, for the first time, a closed-form solution for the stochastic acquisition function optimizer—eliminating the need for Monte Carlo sampling. Contribution/Results: The method substantially reduces computational overhead in noisy environments. Evaluated on two computational engineering numerical experiments, it achieves up to a 40× improvement in iteration efficiency, while simultaneously reducing data requirements and total computational cost by approximately 40×, thereby significantly alleviating resource bottlenecks in hyperparameter tuning.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Hyperparameter tuning is a challenging problem especially when the system itself involves uncertainty. Due to noisy function evaluations, optimization under uncertainty can be computationally expensive. In this paper, we present a novel Bayesian optimization framework tailored for hyperparameter tuning under uncertainty, with a focus on optimizing a scale- or precision-type parameter in stochastic models. The proposed method employs a statistical surrogate for the underlying random variable, enabling analytical evaluation of the expectation operator. Moreover, we derive a closed-form expression for the optimizer of the random acquisition function, which significantly reduces computational cost per iteration. Compared with a conventional one-dimensional Monte Carlo-based optimization scheme, the proposed approach requires 40 times fewer data points, resulting in up to a 40-fold reduction in computational cost. We demonstrate the effectiveness of the proposed method through two numerical examples in computational engineering.

Problem

• Optimizing scale parameters in stochastic models with uncertainty
• Reducing computational cost of hyperparameter tuning under noise
• Developing Bayesian optimization with analytical expectation evaluation

Innovation

• Bayesian optimization framework for hyperparameter tuning under uncertainty
• Statistical surrogate model enables analytical expectation evaluation
• Closed-form optimizer reduces computational cost by 40 times