Multi-objective Evolution of Heuristic Using Large Language Model

2024-09-25

arXiv.org

AI Summary

Existing heuristic search research overemphasizes optimality while neglecting practical requirements such as efficiency and scalability. This paper reformulates heuristic search as a multi-objective optimization problem, jointly optimizing solution quality, runtime efficiency, and scalability. We propose MEoH—a Large Language Model (LLM)-driven Multi-objective Evolutionary framework for Heuristics—that (i) generates heuristic code zero-shot via LLMs; (ii) introduces a dominance-diversity co-guidance mechanism ensuring Pareto-optimality in objective space and preserving semantic diversity in code space; and (iii) evaluates candidates using code similarity metrics and combinatorial optimization benchmarks (BPP/TSP). A single MEoH run yields a diverse set of elite heuristics whose solution quality matches or exceeds handcrafted heuristics, while inference speed improves by up to 10×. Our work establishes a novel paradigm for automated heuristic design grounded in data–model co-optimization.

Technology Category

• Search and Optimization: Heuristic Search
• Constraint Satisfaction and Optimization: Search
• Machine Learning: Optimization

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Heuristics are commonly used to tackle various search and optimization problems. Design heuristics usually require tedious manual crafting with domain knowledge. Recent works have incorporated Large Language Models (LLMs) into automatic heuristic search, leveraging their powerful language and coding capacity. However, existing research focuses on the optimal performance on the target problem as the sole objective, neglecting other criteria such as efficiency and scalability, which are vital in practice. To tackle this challenge, we propose to model the heuristic search as a multi-objective optimization problem and consider introducing additional practical criteria beyond optimal performance. Due to the complexity of the search space, conventional multi-objective optimization methods struggle to effectively handle LLM-based multi-objective heuristic search. We propose the first LLM-based multi-objective heuristic search framework, Multi-objective Evolution of Heuristic (MEoH), which integrates LLMs in a zero-shot manner to generate a non-dominated set of heuristics to meet multiple design criteria. We design a new dominance-dissimilarity mechanism for effective population management and selection, which incorporates both code dissimilarity in the search space and dominance in the objective space. MEoH is demonstrated in two well-known combinatorial optimization problems: the online Bin Packing Problem (BPP) and the Traveling Salesman Problem (TSP). The results indicate that a variety of elite heuristics are automatically generated in a single run, offering more trade-off options than the existing methods. It successfully achieves competitive or superior performance while improving efficiency up to 10 times. Moreover, we also observe that the multi-objective search introduces novel insights into heuristic design and leads to the discovery of diverse heuristics.

Problem

• Optimize heuristic search with multiple objectives
• Enhance efficiency and scalability in heuristic design
• Generate diverse heuristics using Large Language Models

Innovation

• LLM-based multi-objective heuristic search
• zero-shot integration of Large Language Models
• dominance-dissimilarity mechanism for selection