One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models

2026-08-10

AI Summary

This study investigates the nature of knowledge encoded by self-evolution mechanisms in code generation: whether performance gains stem from benchmark-specific adaptation, language-specific expertise, or compensation for model capability gaps. By applying a unified self-evolution pipeline across a grid of eight programming languages and three base models—augmented with type-aware failure signals to guide edits and falsifiable contract logging for attribution—the authors uncover that self-evolution harnesses comprise both cross-language abstract strategies and language-ecosystem-specific implementations. The findings reveal that self-evolution functions primarily as an interpretable compensatory layer, rectifying recoverable execution failures. Moreover, its shared core can be distilled into a general-purpose harness, which significantly improves code-solving success rates across most language–model combinations.

Abstract

Self-evolving harnesses are closed-loop systems in which an agent inspects its own rollouts and edits its prompts, tools, and memory. They reliably improve coding agents in evaluations, but prior work reports aggregate gains rather than analyzing what the evolved artifacts encode. It therefore remains unclear whether they encode benchmark-specific adaptations, language-specific engineering knowledge, or compensation for limitations of the underlying model. We disentangle these factors by holding an evolution recipe fixed across a grid of eight programming languages (Multi-SWE-Bench) and three base models, and analyzing the resulting harnesses. The recipe routes every edit through a typed failure signal and records it as a falsifiable contract, making each modification attributable after evolution. Four findings emerge. (1)The loop improves held-out solve rates over both a minimal seed and the manually designed mini-SWE-agent scaffold in most cells, but with two null regions. (2)Gains compensate recoverable execution defects, where defect mass is near zero, and gain is near zero; which defect dominates is cell-specific. A harness closes the gap between what a policy can do and what it does. (3)Evolved harnesses share an abstract playbook across languages but instantiate it with almost disjoint language ecosystem machinery. (4)The shared core transfers and can be distilled into one universal harness, while an ecosystem margin resists both and requires native re-evolution. Together, these results recast the evolved harness as a legible compensation layer, shaped jointly by the language's engineering demands and the model's behavioral gaps, rather than an opaque benchmark-tuned scaffold.

Problem

• self-evolution
• harnesses
• programming languages
• model limitations
• benchmark adaptation

Innovation

• self-evolving harnesses
• failure signal attribution
• cross-language transfer
• compensation layer
• falsifiable contract