Pitfalls and Remedies for Multi-Task Bayesian Optimization

2026-07-09

AI Summary

This work addresses the frequent failure of multi-task Bayesian optimization due to inaccurate estimation of cross-task correlations—even under the simplified assumption of affine relationships between source and target tasks. The study systematically identifies two root causes: alignment errors induced by task normalization with finite samples and insufficient identifiability of marginal likelihood under non-overlapping experimental designs. To mitigate these issues, the authors propose modeling task-specific means and scales as learnable parameters and introduce three conservative remedies: enforcing non-negative task covariance constraints, adopting partially co-located experimental designs, and refining the correlation estimation mechanism. The method successfully recovers single-task performance on affine synthetic benchmarks and hyperparameter transfer tuning tasks, yet limitations persist in more complex settings, particularly those involving ranking-based objectives or latent contextual structures.

Abstract

Bayesian optimization routinely warm-starts a target experiment with data from related source tasks, and the multi-task Gaussian process is the textbook surrogate for the job. We revisit this default in a controlled setting and find that it misestimates the cross-task correlation even in the simplest non-trivial case, affinely related source and target tasks, where a working transfer learning method should obviously succeed. We trace the failure to two independent structural mechanisms. Per-task standardization, the textbook fix for the affine slice ambiguity, propagates a finite-sample alignment error into the recovered correlation. The marginal likelihood itself identifies the correlation only at a per-sample rate that a Gaussian process at non-overlapping designs further dilutes. We propose three conservative remedies that follow from the analysis: promoting per-task means and scales to model parameters, restricting the task covariance to non-negative correlations, and co-locating part of the source and target designs. Across synthetic multi-task problems and surrogate-based hyperparameter tuning transfer, these remedies recover the target-only baseline on the simple instances, while the broader failure persists on harder instances and across most rank-based and latent-context variants.

Problem

• multi-task Bayesian optimization
• cross-task correlation
• Gaussian process
• transfer learning
• affine tasks

Innovation

• multi-task Bayesian optimization
• Gaussian process
• transfer learning
• task correlation
• experimental design