Transfer Learning of Surrogate Models via Domain Affine Transformation Across Synthetic and Real-World Benchmarks

2025-01-23

AI Summary

Surrogate modeling in black-box optimization suffers from high training costs and scarcity of target-domain data. Method: We propose the first domain-affine transfer framework tailored for non-differentiable surrogates (e.g., random forests), which operates without gradient information by explicitly modeling affine relationships between source and target domains in function space. Contribution/Results: Evaluated on the BBOB benchmark and four real-world black-box optimization tasks, our method achieves significant reductions in data requirements and computational overhead—requiring only a few target-domain evaluations—while improving surrogate modeling efficiency by 2–5×. It demonstrates strong robustness and generalization across diverse problem settings. To our knowledge, this is the first work to extend domain adaptation principles to non-differentiable surrogate models, establishing a practical, low-budget paradigm for black-box optimization.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Surrogate models are frequently employed as efficient substitutes for the costly execution of real-world processes. However, constructing a high-quality surrogate model often demands extensive data acquisition. A solution to this issue is to transfer pre-trained surrogate models for new tasks, provided that certain invariances exist between tasks. This study focuses on transferring non-differentiable surrogate models (e.g., random forest) from a source function to a target function, where we assume their domains are related by an unknown affine transformation, using only a limited amount of transfer data points evaluated on the target. Previous research attempts to tackle this challenge for differentiable models, e.g., Gaussian process regression, which minimizes the empirical loss on the transfer data by tuning the affine transformations. In this paper, we extend the previous work to the random forest model and assess its effectiveness on a widely-used artificial problem set - Black-Box Optimization Benchmark (BBOB) testbed, and on four real-world transfer learning problems. The results highlight the significant practical advantages of the proposed method, particularly in reducing both the data requirements and computational costs of training surrogate models for complex real-world scenarios.

Problem

• Transfer Learning
• Data Efficiency
• Computational Cost

Innovation

• Transfer Learning
• Random Forest
• Data Efficiency