Problem
Research questions and friction points this paper is trying to address.
AI programming
software reimplementation
long-horizon coding
autonomous coding
benchmarking
MirrorCode: AI can rebuild entire programs from behavior alone
🤖 AI Summary
Current AI programming evaluations are largely confined to short, isolated tasks and lack standardized benchmarks for assessing the ability to reproduce complete software systems end-to-end. This work proposes MirrorCode, a novel long-horizon programming benchmark that introduces a behavior-based reverse-engineering paradigm: under strict black-box conditions without access to source code, AI systems must precisely reconstruct the functionality of real-world software solely from its observable behavior. The benchmark encompasses 25 complex projects spanning Unix utilities, compilers, and bioinformatics tools. Leveraging large language model–driven autonomous agents and a high-fidelity test-validation framework, the strongest model achieves a 56% success rate, including the successful reproduction of large-scale tools such as gotree (16,000 lines), demonstrating a significant leap in AI’s capacity for complex software engineering tasks.
📝 Abstract
AI models are rapidly improving at autonomous coding, as shown by benchmark progress and one-off demonstrations such as AI implementing a C compiler. However, existing coding benchmarks tend to focus on shorter tasks, and one-off demonstrations are hard to compare systematically because they often have some human guidance, and are not standardized or repeated across models. To address these challenges, we introduce MirrorCode, a long-horizon coding benchmark based on reimplementing entire software projects. In MirrorCode, AI agents must replicate the functionalities of an existing program, without access to its source code. AI solutions must match the original program's output exactly on end-to-end tests, including held-out tests. MirrorCode's 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression. Existing AI models can already reimplement complex software, with the strongest model scoring 56% across the benchmark. For example, AI can reimplement gotree, a 16,000-line bioinformatics toolkit - a task that we believe would take weeks for a human engineer. However, studying the frontier of performance requires a larger inference budget than typical benchmarks, for example, \$2,600 over 19 days for a single attempt on a large task. We show that AI agents can already complete long-horizon software engineering tasks, especially when requirements are precisely specified. More broadly, our work suggests AI will have transformative effects on software engineering, as autonomous agents continue to improve.
Innovation
Methods, ideas, or system contributions that make the work stand out.
MirrorCode
long-horizon coding
program synthesis from behavior
autonomous software engineering
end-to-end program replication
🔎 Similar Papers
No similar papers found.
