Surrogate Ensemble in Expensive Multi-Objective Optimization via Deep Q-Learning

2026-01-31

AI Summary

This work addresses the challenge in expensive multi-objective optimization where manually selecting surrogate models introduces bias and limits algorithmic generalization. To overcome this, we propose SEEMOO, a novel framework that, for the first time, integrates deep Q-learning into dynamic surrogate model scheduling. SEEMOO features an adaptive ensemble mechanism composed of a predefined surrogate pool, an attention-based state extractor, and a deep Q-network, enabling the collaborative fusion of multiple surrogate models within a single optimization run without human intervention. Experimental results demonstrate that SEEMOO consistently outperforms single-surrogate approaches across multiple benchmark problems, while ablation studies confirm the effectiveness of each component in the proposed architecture.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Learning & Optimization for ROB
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Surrogate-assisted Evolutionary Algorithms~(SAEAs) have shown promising robustness in solving expensive optimization problems. A key aspect that impacts SAEAs'effectiveness is surrogate model selection, which in existing works is predominantly decided by human developer. Such human-made design choice introduces strong bias into SAEAs and may hurt their expected performance on out-of-scope tasks. In this paper, we propose a reinforcement learning-assisted ensemble framework, termed as SEEMOO, which is capable of scheduling different surrogate models within a single optimization process, hence boosting the overall optimization performance in a cooperative paradigm. Specifically, we focus on expensive multi-objective optimization problems, where multiple objective functions shape a compositional landscape and hence challenge surrogate selection. SEEMOO comprises following core designs: 1) A pre-collected model pool that maintains different surrogate models; 2) An attention-based state-extractor supports universal optimization state representation of problems with varied objective numbers; 3) a deep Q-network serves as dynamic surrogate selector: Given the optimization state, it selects desired surrogate model for current-step evaluation. SEEMOO is trained to maximize the overall optimization performance under a training problem distribution. Extensive benchmark results demonstrate SEEMOO's surrogate ensemble paradigm boosts the optimization performance of single-surrogate baselines. Further ablation studies underscore the importance of SEEMOO's design components.

Problem

• expensive multi-objective optimization
• surrogate model selection
• surrogate-assisted evolutionary algorithms
• optimization performance
• model bias

Innovation

• Surrogate Ensemble
• Deep Q-Learning
• Multi-Objective Optimization
• Attention-based State Representation
• Reinforcement Learning-assisted Selection