LLM-Driven Evolutionary Generation of Multi-Objective Bayesian Optimization Algorithms

2026-07-06

AI Summary

This work proposes a novel framework that leverages large language models (LLMs) to automatically design multi-objective Bayesian optimization (MOBO) algorithms, addressing the heavy reliance on expert knowledge and the difficulty of coordinating complex algorithmic components. For the first time, an LLM is employed as both mutation and crossover operators within an evolutionary strategy to end-to-end generate complete MOBO algorithms. The framework integrates SMAC for hyperparameter optimization within the evolutionary loop, operating without human intervention. Evaluated on standard benchmarks (ZDT, DTLZ, WFG, RE) and real-world engineering problems, the generated algorithms achieve superior average normalized hypervolume compared to the qParEGO baseline—significantly outperforming it on 7 out of 12 problems without ever being worse—and demonstrate computational speedups ranging from 3.4× to 60×.

Abstract

Designing effective multi-objective Bayesian optimization (MOBO) algorithms requires balancing many interdependent design choices whose optimal configuration is problem-dependent and typically demands deep expertise. We extend the LLaMEA framework to MOBO, using large language models as mutation and crossover operators within evolutionary strategies to generate complete algorithm implementations, with SMAC hyperparameter optimization integrated into the evolutionary loop. Across nine evolutionary runs we generated approximately 900 algorithms and benchmarked them on twelve synthetic problems (ZDT, DTLZ, WFG) and three real-world engineering problems (RE), using a BoFire qParEGO implementation as a state-of-the-art Bayesian-optimization baseline. On the synthetic suite the strongest generated algorithm attains the highest mean normalized hypervolume (0.971, vs. 0.869 for qParEGO) while requiring roughly 60x less wall-clock time; a Friedman test with post-hoc analysis places the two in a single top-performing group, and per-problem tests find the generated algorithm significantly better than qParEGO on 7 of the 12 problems and never worse, matching state-of-the-art accuracy at an order-of-magnitude lower cost. On the three unseen real-world engineering problems a generated algorithm attains the best mean normalized hypervolume (0.985, vs. 0.971 for qParEGO)--significantly better than qParEGO on two of the three problems--at roughly 3.4x lower wall-clock cost, confirming that the gains transfer beyond the synthetic regime. LLM-driven evolutionary search can thus discover algorithm designs that achieve Pareto-efficient trade-offs difficult to reach through manual design.

Problem

• Multi-objective Bayesian Optimization
• Algorithm Design
• Pareto Efficiency
• Hyperparameter Optimization
• Evolutionary Search

Innovation

• LLM-driven evolution
• multi-objective Bayesian optimization
• algorithm generation
• evolutionary strategies
• automated algorithm design