AppForge: From Assistant to Independent Developer - Are GPTs Ready for Software Development?

2025-10-08

AI Summary

This work investigates the capability of large language models (LLMs) to generate complete, functional Android applications from scratch, with particular emphasis on system-level challenges—including state coordination, activity lifecycle management, and asynchronous operations. To enable rigorous evaluation, we introduce AppForge, the first automated, end-to-end benchmark for Android application generation, comprising 101 development tasks derived from real-world applications. We propose a novel multi-agent framework that autonomously generates functional specifications and executable test cases, all validated by domain experts to ensure fidelity and reproducibility. Extensive evaluation across 12 state-of-the-art LLMs reveals severe limitations: even the strongest model, GPT-5, achieves only 18.8% task success rate. These results expose fundamental shortcomings of current LLMs in handling complex, structured software engineering tasks requiring deep platform-specific knowledge and precise compositional reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Multiagent Systems: Applications

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) have demonstrated remarkable capability in function-level code generation tasks. Unlike isolated functions, real-world applications demand reasoning over the entire software system: developers must orchestrate how different components interact, maintain consistency across states over time, and ensure the application behaves correctly within the lifecycle and framework constraints. Yet, no existing benchmark adequately evaluates whether LLMs can bridge this gap and construct entire software systems from scratch. To address this gap, we propose APPFORGE, a benchmark consisting of 101 software development problems drawn from real-world Android apps. Given a natural language specification detailing the app functionality, a language model is tasked with implementing the functionality into an Android app from scratch. Developing an Android app from scratch requires understanding and coordinating app states, lifecycle management, and asynchronous operations, calling for LLMs to generate context-aware, robust, and maintainable code. To construct APPFORGE, we design a multi-agent system to automatically summarize the main functionalities from app documents and navigate the app to synthesize test cases validating the functional correctness of app implementation. Following rigorous manual verification by Android development experts, APPFORGE incorporates the test cases within an automated evaluation framework that enables reproducible assessment without human intervention, making it easily adoptable for future research. Our evaluation on 12 flagship LLMs show that all evaluated models achieve low effectiveness, with the best-performing model (GPT-5) developing only 18.8% functionally correct applications, highlighting fundamental limitations in current models' ability to handle complex, multi-component software engineering challenges.

Problem

• Evaluating LLMs' ability to build complete software systems from scratch
• Addressing the gap in benchmarks for full application development
• Testing model performance on real-world Android app creation challenges

Innovation

• Proposed APPFORGE benchmark for full software development
• Used multi-agent system to synthesize test cases
• Created automated evaluation framework without human intervention