Optimization with SpotOptim

2026-04-15

AI Summary

This work addresses high-cost black-box optimization problems involving mixed-variable types, noisy evaluations, and multiple objectives by proposing a surrogate modeling framework grounded in Sequential Parameter Optimization (SPO). The approach integrates Kriging models with expected improvement criteria, accommodating continuous, integer, and categorical variables, while incorporating Optimal Computing Budget Allocation (OCBA) to handle noise. It introduces a novel steady-state parallelization strategy that overlaps surrogate-based search with expensive true evaluations and employs a success-rate-based restart mechanism to mitigate premature convergence. Designed with compatibility for scikit-learn interfaces and real-time monitoring via TensorBoard, the framework demonstrates competitive or superior performance compared to state-of-the-art libraries such as BoTorch, Optuna, and Ray Tune on tasks including neural network hyperparameter tuning. The implementation is publicly released.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

The `spotoptim` package implements surrogate-model-based optimization of expensive black-box functions in Python. Building on two decades of Sequential Parameter Optimization (SPO) methodology, it provides a Kriging-based optimization loop with Expected Improvement, support for continuous, integer, and categorical variables, noise-aware evaluation via Optimal Computing Budget Allocation (OCBA), and multi-objective extensions. A steady-state parallelization strategy overlaps surrogate search with objective evaluation on multi-core hardware, and a success-rate-based restart mechanism detects stagnation while preserving the best solution found. The package returns scipy-compatible `OptimizeResult` objects and accepts any scikit-learn-compatible surrogate model. Built-in TensorBoard logging provides real-time monitoring of convergence and surrogate quality. This report describes the architecture and module structure of spotoptim, provides worked examples including neural network hyperparameter tuning, and compares the framework with BoTorch, Optuna, Ray Tune, BOHB, SMAC, and Hyperopt. The package is open-source.

Problem

• black-box optimization
• expensive functions
• surrogate modeling
• multi-objective optimization
• hyperparameter tuning

Innovation

• surrogate-model-based optimization
• Sequential Parameter Optimization (SPO)
• noise-aware evaluation
• steady-state parallelization
• multi-objective optimization