What Do Evolutionary Coding Agents Evolve?

2026-05-19

AI Summary

Existing evolutionary coding systems excel in mathematical and algorithmic tasks, yet the mechanisms driving their performance gains—such as novel algorithmic structures, hyperparameter tuning, or repeated code edits—remain poorly understood. This work introduces the EvoTrace dataset and the EvoReplay replay analysis framework to systematically disentangle and quantify distinct improvement mechanisms. By conducting fine-grained tracing and controlled interventions—including LLM-based annotations, human blind evaluations, and ablation studies—on code edit trajectories from four prominent evolutionary frameworks across 16 tasks, we find that most score improvements stem from a small set of edit types. Notably, nearly 30% of newly added code consists of exact repetitions of previously deleted lines, revealing deterministic cyclic patterns. These findings indicate that benchmark score increases do not necessarily reflect genuine algorithmic innovation, and our approach provides diagnostic evaluation capabilities for evolutionary coding agents beyond final performance metrics.

Abstract

Recent work pairs LLMs with evolutionary search to iteratively generate, modify, and select code using task-specific feedback. These systems have produced strong results in mathematical discovery and algorithm design, yet a fundamental question remains: what do they actually evolve? Progress is typically summarized by the best score a run reaches under a task-specific evaluator, but that score can reflect several different mechanisms: new algorithmic structure, re-tuning an existing strategy, recombining ideas already in the model's internal knowledge, or overfitting to the evaluator. Distinguishing these mechanisms requires inspecting the search process itself, not only its final outcome. We introduce EvoTrace, a dataset of evolutionary coding traces spanning four evolutionary frameworks, reasoning and non-reasoning models, and 16 tasks across mathematics and algorithm design. To analyze these traces, we develop EvoReplay, a replay-based methodology that reconstructs the local search states behind high-scoring solutions and tests controlled interventions, including adjusting constants, removing program components and substituting models or prompting contexts. We annotate every code edit in EvoTrace with one of nine recurring edit types using an LLM-as-judge pipeline validated against blind human re-annotation. Across EvoTrace, most score gains come from a small subset of these edit types. We further find a deterministic cycling pattern: about 30% of code lines added during search are byte-identical re-introductions of previously-deleted lines, present throughout nearly every run. These results show that benchmark gains in evolutionary coding agents can arise from qualitatively different mechanisms, only some of which correspond to new algorithmic structure. EvoTrace enables more diagnostic evaluation of evolutionary coding agents beyond final benchmark scores.

Problem

• evolutionary coding agents
• algorithmic structure
• code evolution
• evaluation mechanisms
• LLM-based search

Innovation

• evolutionary coding agents
• EvoTrace
• EvoReplay
• code evolution analysis
• LLM-as-judge