Code Evolution Graphs: Understanding Large Language Model Driven Design of Algorithms

2025-03-20

AI Summary

This paper addresses the stagnation in performance, code redundancy, and stylistic rigidity commonly observed when large language models (LLMs) iteratively generate algorithms within evolutionary computation frameworks. To model the dynamic iterative trajectory of LLM-generated code, we propose the **Code Evolution Graph**—the first formal graph-based representation of such evolution. Leveraging static analysis, graph representation learning, and cross-model behavioral comparison across three benchmark task categories, we empirically reveal: (i) iterative generation often increases code complexity while degrading performance; (ii) generated code exhibits significant heterogeneity and stylistic isolation across models; and (iii) repeated prompting induces redundant overcomplication. Building on these insights, we introduce a **multi-LLM co-evolution paradigm**, which demonstrably mitigates degeneration and improves solution quality. Our approach provides an interpretable, controllable pathway for LLM-driven automated algorithm design.

Technology Category

• Machine Learning: Evolutionary Learning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Evolutionary Computation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have demonstrated great promise in generating code, especially when used inside an evolutionary computation framework to iteratively optimize the generated algorithms. However, in some cases they fail to generate competitive algorithms or the code optimization stalls, and we are left with no recourse because of a lack of understanding of the generation process and generated codes. We present a novel approach to mitigate this problem by enabling users to analyze the generated codes inside the evolutionary process and how they evolve over repeated prompting of the LLM. We show results for three benchmark problem classes and demonstrate novel insights. In particular, LLMs tend to generate more complex code with repeated prompting, but additional complexity can hurt algorithmic performance in some cases. Different LLMs have different coding ``styles'' and generated code tends to be dissimilar to other LLMs. These two findings suggest that using different LLMs inside the code evolution frameworks might produce higher performing code than using only one LLM.

Problem

• Understanding LLM-driven algorithm design evolution
• Analyzing code evolution in iterative LLM prompting
• Comparing coding styles across different LLM models

Innovation

• Analyze code evolution via repeated LLM prompting
• Compare coding styles across different LLMs
• Combine multiple LLMs to enhance performance