Advancing Automated Algorithm Design via Evolutionary Stagewise Design with LLMs

2026-03-09

AI Summary

This work addresses the limitations of traditional algorithm design in tackling complex industrial problems and the inadequacy of existing large language model (LLM)-based automated approaches, which often yield inefficient or unreasonable solutions due to their black-box nature and lack of mechanistic understanding. To overcome these challenges, we propose EvoStage, a novel framework that integrates a staged evolutionary mechanism with LLMs, inspired by chain-of-thought reasoning. EvoStage decomposes algorithm design into a multi-stage process, incorporating multi-agent collaboration and a hybrid global-local optimization strategy, with iterative refinement driven by real-time intermediate feedback. This approach substantially reduces the search space and mitigates local optima, achieving state-of-the-art results: in chip placement and Bayesian optimization tasks, it surpasses both human experts and current LLM-based methods within just a few evolutionary rounds, attaining the best-known half-perimeter wirelength across all benchmarks and setting a new performance record in commercial 3D chip placement tools.

Technology Category

• Search and Optimization: Evolutionary Computation
• Machine Learning: Evolutionary Learning
• Multiagent Systems: Mechanism Design

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

With the rapid advancement of human science and technology, problems in industrial scenarios are becoming increasingly challenging, bringing significant challenges to traditional algorithm design. Automated algorithm design with LLMs emerges as a promising solution, but the currently adopted black-box modeling deprives LLMs of any awareness of the intrinsic mechanism of the target problem, leading to hallucinated designs. In this paper, we introduce Evolutionary Stagewise Algorithm Design (EvoStage), a novel evolutionary paradigm that bridges the gap between the rigorous demands of industrial-scale algorithm design and the LLM-based algorithm design methods. Drawing inspiration from CoT, EvoStage decomposes the algorithm design process into sequential, manageable stages and integrates real-time intermediate feedback to iteratively refine algorithm design directions. To further reduce the algorithm design space and avoid falling into local optima, we introduce a multi-agent system and a"global-local perspective"mechanism. We apply EvoStage to the design of two types of common optimizers: designing parameter configuration schedules of the Adam optimizer for chip placement, and designing acquisition functions of Bayesian optimization for black-box optimization. Experimental results across open-source benchmarks demonstrate that EvoStage outperforms human-expert designs and existing LLM-based methods within only a couple of evolution steps, even achieving the historically state-of-the-art half-perimeter wire-length results on every tested chip case. Furthermore, when deployed on a commercial-grade 3D chip placement tool, EvoStage significantly surpasses the original performance metrics, achieving record-breaking efficiency. We hope EvoStage can significantly advance automated algorithm design in the real world, helping elevate human productivity.

Problem

• automated algorithm design
• large language models
• industrial-scale optimization
• black-box modeling
• algorithm hallucination

Innovation

• Evolutionary Stagewise Design
• LLM-based Algorithm Design
• Multi-agent System
• Global-local Perspective
• Automated Algorithm Design