From I/O to Code with Discovery Agent

2026-05-14

AI Summary

This work addresses the long-standing challenge of input-output example-to-code synthesis (IO2Code), which is hindered by semantic misalignment and an expansive search space. The authors propose a novel evolutionary search method that leverages large language models as mutation operators, guided by program execution feedback. Central to their approach is the Transformation Priority Premise—a mutation prior that prioritizes exploring structurally simpler hypotheses in a principled order (e.g., constants before conditionals, and conditionals before loops). Evaluated on a newly introduced IO2CodeBench benchmark, the method consistently outperforms conventional program synthesis techniques, state-of-the-art evolutionary agents, and test-time scaling strategies across all difficulty levels and multiple large language models under equivalent sampling budgets.

Abstract

The automatic synthesis of a program from any form of specification is regarded as a holy grail of computer science. Fueled by LLMs, NL2Code has achieved tremendous success, yet the fundamentally more challenging task of synthesizing programs from input-output behavior, which we refer to as IO2Code, remains largely unsolved. Whereas NL2Code can exploit the semantic alignment between natural language and code acquired during pretraining, IO2Code requires recovering underlying principles from concrete computational behavior, navigating a vast and underspecified hypothesis space. To address this, we propose DIO-Agent, a discovery agent for IO2Code. Our method frames IO2Code as an evolutionary search over discrete program space, in which an LLM serves as the mutation operator and concrete error signals from execution guide each mutation. To prevent the search from wandering into structurally complex yet incorrect dead ends, we introduce the Transformation Priority Premise as a mutation prior that biases the LLM toward the simplest hypothesis consistent with current evidence, progressively escalating from constants to conditionals to iteration only when simpler constructs are insufficient. To facilitate systematic study, we further construct an IO2CodeBench spanning multiple difficulty levels. Extensive experiments show that DIO-Agent consistently outperforms both traditional program-by-example method and SOTA evolution-agent baselines across all difficulty levels and various LLMs, while substantially surpassing test-time scaling strategies with equivalent sampling budgets.

Problem

• IO2Code
• program synthesis
• input-output behavior
• automatic programming
• code generation

Innovation

• IO2Code
• program synthesis
• evolutionary search
• large language models
• Transformation Priority Premise