AHD Agent: Agentic Reinforcement Learning for Automatic Heuristic Design

2026-05-09

AI Summary

Existing automated heuristic design methods are constrained by fixed pipelines and struggle to leverage state-dependent information during the solving process, resulting in inefficient exploration. This work proposes the AHD Agent framework, which introduces an active agent mechanism into this domain for the first time. By integrating tool use, multi-turn interaction, and environment synthesis, the framework enables large language models to dynamically decide whether to generate heuristics or acquire environmental evidence, with their decision-making policies trained via reinforcement learning. Experiments across eight domains—including four unseen tasks—demonstrate that a 4B-parameter model achieves performance on par with or superior to larger baselines while significantly reducing the number of required evaluations, thereby validating the approach’s advantages in generalization and sample efficiency.

Abstract

Automatic heuristic design (AHD) has emerged as a promising paradigm for solving NP-hard combinatorial optimization problems (COPs). Recent works show that large language models (LLMs), when integrated into well-designed frameworks (i.e., LLM-AHD), can autonomously discover high-performing heuristics. However, existing LLM-AHD frameworks typically treat LLMs as passive generators within fixed workflows, where the model generates heuristics from manually designed, limited context. Such context may fail to capture state-dependent information (e.g., specific failure modes), leading to inefficient trial-and-error exploration. To overcome these limitations, we propose AHD Agent, a novel tool-integrated, multi-turn framework that empowers LLMs to proactively decide whether to generate heuristics or invoke tools to retrieve targeted evidence from the solving environment. To effectively train such a dynamic decision-making agent, we introduce an agentic reinforcement learning (RL) system, which leverages a novel environment synthesis pipeline to optimize a compact model's generalizable AHD capabilities. Experiments across eight diverse domains, including four held-out tasks, demonstrate that our 4B-parameter agent matches or surpasses state-of-the-art baselines using much larger models, while requiring significantly fewer evaluations. Model and inference scaling analysis further reveals that AHD Agent offers an effective trajectory toward truly autonomous heuristic design.

Problem

• Automatic Heuristic Design
• Combinatorial Optimization Problems
• Large Language Models
• State-dependent Information
• Trial-and-error Exploration

Innovation

• Agentic Reinforcement Learning
• Automatic Heuristic Design
• Tool-Integrated LLM
• Environment Synthesis
• Combinatorial Optimization