BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services

2026-07-12

AI Summary

This study evaluates whether large language models can generate deployable, end-to-end backend services that behave correctly. To this end, the authors construct a benchmark comprising 56 tasks derived from real-world open-source projects, requiring models to produce Dockerizable services conforming to OpenAPI specifications and validated through HTTP black-box testing. The work introduces an innovative co-evolution mechanism between test agents and code agents, which strengthens evaluation rigor without introducing hidden requirements, thereby exposing a significant gap between current models’ ability to implement individual APIs and their capacity to construct complete, functional services. Experimental results show that even the strongest model, GPT-5.5, achieves only a 55.4% success rate on the base test set and drops to 28.6% on the more stringent final test set, underscoring the limitations of existing models in reliably generating full backend systems.

Abstract

Large language models (LLMs) are increasingly used in agentic coding settings, where they can inspect files, execute commands, run tests, observe failures, and iteratively revise code. This shift raises a central evaluation question: can an agentic LLM generate an end-to-end software artifact that is both deployable and behaviorally correct under execution? Backend services provide a controlled but realistic substrate for this evaluation. Their APIs expose application-level executable semantics, and deployed behavior can be checked deterministically against an OpenAPI contract through black-box HTTP interactions. We introduce BackendForge, a benchmark of 56 contract-defined backend generation tasks rewritten from real open-source applications. Given a visible specification and an OpenAPI contract, an LLM must generate a Dockerized service that is built, deployed, and evaluated only through HTTP tests. To strengthen evaluation without introducing hidden requirements, BackendForge uses a test agent and a code agent to co-evolve the test oracle and reference service, where the test agent proposes specification-grounded backend tests and the code agent repairs the reference implementation. Although the best-performing model, GPT-5.5, succeeds on 55.4\% of tasks under the base oracle, it succeeds on only 28.6\% under the final oracle. This gap suggests that current LLMs can implement many local API behaviors, but still struggle to produce complete backend services.

Problem

• agentic code generation
• backend services
• end-to-end code generation
• LLM evaluation
• OpenAPI contract

Innovation

• agentic code generation
• backend service benchmarking
• OpenAPI contract
• co-evolutionary evaluation
• end-to-end software generation