An Empirical Study on Ensemble-Based Transfer Learning Bayesian Optimisation with Mixed Variable Types

2026-01-22

AI Summary

This work addresses the limited sample efficiency of Bayesian optimization in expensive black-box function optimization by proposing an ensemble-based transfer learning approach that effectively leverages relevant historical data to enhance optimization performance. The method integrates predictions from multiple surrogate models using a positivity-constrained weighting strategy grounded in regularized regression and incorporates a mechanism for detecting and responding to negative transfer. Key contributions include the introduction of a positivity-constrained ensemble weighting scheme coupled with a robust warm-starting mechanism, the construction of three new benchmarks specifically designed for real-time transfer learning in Bayesian optimization, and comprehensive experimental validation demonstrating that the proposed approach significantly outperforms existing methods while maintaining strong reproducibility.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Bayesian optimisation is a sample efficient method for finding a global optimum of expensive black-box objective functions. Historic datasets from related problems can be exploited to help improve performance of Bayesian optimisation by adapting transfer learning methods to various components of the Bayesian optimisation pipeline. In this study we perform an empirical analysis of various ensemble-based transfer learning Bayesian optimisation methods and pipeline components. We expand on previous work in the literature by contributing some specific pipeline components, and three new real-time transfer learning Bayesian optimisation benchmarks. In particular we propose to use a weighting strategy for ensemble surrogate model predictions based on regularised regression with weights constrained to be positive, and a related component for handling the case when transfer learning is not improving Bayesian optimisation performance. We find that in general, two components that help improve transfer learning Bayesian optimisation performance are warm start initialisation and constraining weights used with ensemble surrogate model to be positive.

Problem

• Bayesian optimisation
• transfer learning
• ensemble methods
• mixed variable types
• black-box optimisation

Innovation

• ensemble-based transfer learning
• Bayesian optimisation
• positive weight constraint
• warm start initialisation
• mixed variable types