How to Guide LLM Generation: Dual-Surrogate Guided Search for Automated Heuristic Design

2026-07-15

AI Summary

This work addresses the inefficiency in action selection when large language models (LLMs) automatically generate heuristics under limited query and evaluation budgets. To overcome this challenge, the authors propose a dual-agent guided action selection mechanism: a transition agent models the generative distribution of latent representations of sub-heuristics, while a utility agent predicts their performance on specific problem instances. An acquisition strategy that integrates the uncertainties of both agents dynamically guides the selection of parent heuristics and generation operators. This approach transcends the limitations of traditional rule-based or static-preference methods, achieving performance that matches or surpasses strong existing baselines across multiple heuristic design tasks. Ablation studies further confirm the effectiveness and non-trivial contribution of the proposed mechanism.

Abstract

Large language models (LLMs) have made automated heuristic design (AHD) increasingly practical by generating executable heuristic code from task descriptions and evaluator feedback. Yet under a limited query and evaluation budget, search efficiency depends critically on a pre-generation decision. Before each LLM query and black-box evaluation, the system must choose which archived heuristics to reuse as parents and which generation operator should transform them. Existing methods typically choose such actions with predefined rules, leaving the expected outcome of each concrete operator-parent action only indirectly modeled. Therefore, we propose \emph{\fullmethod{}} (\method{}), a surrogate-guided action-selection module for operator-parent selection in LLM-based AHD. \method{} guides the LLM code-generation process by scoring pre-generation actions with two complementary surrogates. Specifically, a transition surrogate is proposed to predict the latent distribution of the child representation induced by an operator-parent action, while an instance-conditioned utility surrogate is proposed to estimate the expected performance of sampled child latents. Moreover, we propose an uncertainty-aware acquisition rule that combines predicted utility, utility uncertainty, and transition uncertainty to select the next LLM generation action. Across a diverse heuristic-design suite, \method{} is competitive with strong LLM-AHD baselines, and ablation and action-selection analyses suggest that its behavior goes beyond simple archive ranking or fixed operator preferences.

Problem

• automated heuristic design
• large language models
• operator-parent selection
• search efficiency
• surrogate modeling

Innovation

• surrogate-guided search
• automated heuristic design
• large language models
• uncertainty-aware acquisition
• latent representation prediction