ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development

2026-01-16

AI Summary

Existing code generation benchmarks primarily focus on static logic and fail to evaluate agents’ ability to handle end-to-end tasks in real-world backend development, such as environment configuration and service deployment. To address this gap, this work proposes the first executable evaluation benchmark that spans the full lifecycle of backend engineering, requiring agents to comprehend codebases, implement solutions across eight programming languages and nineteen frameworks, containerize services, and validate APIs end-to-end. The benchmark leverages an automated pipeline to extract tasks from open-source projects and employs containerization with external testing to ensure realistic and reproducible evaluation. Experimental results demonstrate that even state-of-the-art large language models perform poorly on this benchmark, revealing a significant capability gap in practical software engineering scenarios.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The evolution of Large Language Models (LLMs) into autonomous agents has expanded the scope of AI coding from localized code generation to complex, repository-level, and execution-driven problem solving. However, current benchmarks predominantly evaluate code logic in static contexts, neglecting the dynamic, full-process requirements of real-world engineering, particularly in backend development which demands rigorous environment configuration and service deployment. To address this gap, we introduce ABC-Bench, a benchmark explicitly designed to evaluate agentic backend coding within a realistic, executable workflow. Using a scalable automated pipeline, we curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories. Distinct from previous evaluations, ABC-Bench require the agents to manage the entire development lifecycle from repository exploration to instantiating containerized services and pass the external end-to-end API tests. Our extensive evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks, highlighting a substantial disparity between current model capabilities and the demands of practical backend engineering. Our code is available at https://github.com/OpenMOSS/ABC-Bench.

Problem

• backend development
• code generation
• software engineering
• LLM agents
• executable evaluation

Innovation

• agentic coding
• backend development
• executable benchmark
• end-to-end evaluation
• containerized deployment