Back to the Beginning of Heuristic Design: Bridging Code and Knowledge with LLMs

2026-05-07

AI Summary

This work addresses the limitations of existing automated heuristic design approaches, which predominantly rely on bottom-up code search and struggle to extract reusable, transferable high-level knowledge. The authors propose a top-down, knowledge-first search paradigm that treats knowledge as the primary object of search, using code merely for instantiation and validation. By formulating explainable hypotheses, the method enables knowledge reuse across problems and solution trajectories. It establishes, for the first time, a bidirectional bridge between knowledge and code, introduces a statistical learning perspective to characterize the distortion–compression trade-off, and integrates large language models with population-based and tree search mechanisms into a unified, knowledge-driven iterative optimization framework. Experiments demonstrate that this approach significantly outperforms code-centric methods in heuristic discovery efficiency, transferability, and generalization across combinatorial optimization and extended tasks.

Abstract

Large language models (LLMs) have recently advanced automatic heuristic design (AHD) for combinatorial optimization (CO), where candidate heuristics are iteratively proposed, evaluated, and refined. Most existing approaches search over executable programs and distill insights from execution feedback to guide later iterations. Because this process moves from low-level implementations to high-level principles, we refer to it as a bottom-up paradigm. We argue that this view is incomplete and introduce a complementary top-down perspective: knowledge becomes the primary search object and code merely instantiates and tests it, making what is learned explicit and reusable across problems and trajectories. We formalize this shift through a statistical-learning view that exposes a distortion--compression trade-off, and instantiate it in both population-based and tree-based AHD frameworks. Across CO and tasks beyond it, knowledge-first search improves discovery efficiency, transfer, and generalization, often outperforming code-centric pipelines, while combining both strategies yields further gains. Our results suggest that progress in AHD depends on iteratively constructing and evolving interpretable hypotheses that retain value beyond a single search trajectory.

Problem

• automatic heuristic design
• combinatorial optimization
• large language models
• knowledge representation
• search paradigm

Innovation

• knowledge-first search
• automatic heuristic design
• combinatorial optimization
• top-down paradigm
• interpretable hypotheses