From What to How: Bridging User Requirements with Software Development Using Large Language Models

2026-02-14

AI Summary

This work addresses the limited support of current large language models (LLMs) for the software design phase, where accurately translating requirements into implementable designs remains a challenge. To bridge this gap, we introduce DesBench, the first evaluation benchmark specifically tailored for software design, comprising 30 manually curated Java projects that include requirements documents, design models, implementation code, and test cases. Using DesBench, we systematically assess mainstream LLMs across three key tasks: design-aware code generation, object-oriented modeling, and acceptance test design. Our findings reveal that LLMs struggle to produce correct code when no design or only high-level design is provided, exhibit significant deficiencies in modeling class relationships, yet generate acceptance tests that achieve code coverage comparable to human-written ones.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recently, large language models (LLMs) are extensively utilized to enhance development efficiency, leading to numerous benchmarks for evaluating their performance. However, these benchmarks predominantly focus on implementation, overlooking the equally critical aspect of software design. This gap raises two pivotal questions: (1) Can LLMs handle software design? (2) Can LLMs write code following the specific designs? To investigate these questions, this paper proposes DesBench, a design-aware benchmark for evaluating LLMs on three software design-related tasks: design-aware code generation, object-oriented modeling, and the design of acceptance test cases. DesBench comprises 30 manually crafted Java projects that include requirement documents, design models, implementations, and acceptance tests, amounting to a total of 30 design models, 194 Java classes, and 737 test cases. We evaluated seven state-of-the-art LLMs, including three DeepSeek R1, two Qwen2.5, and two GPT models, using DesBench. The results reveal that LLMs remain significantly challenged by the intricacies of software design: (1) For code generation, LLMs struggle to produce correct implementations when provided with only high-level or no designs. (2) In object-oriented modeling, while LLMs can accurately identify objects and classes, they face challenges in defining operations and inter-class relationships. (3) Acceptance test cases generated by LLMs from functional requirements achieve code coverage quality comparable to those written by humans. Our research highlights the current limitations of LLMs in managing software design and calls for further investigation into new design methodologies and languages suitable for LLM-based development.

Problem

• software design
• large language models
• code generation
• object-oriented modeling
• acceptance testing

Innovation

• design-aware benchmark
• large language models
• software design
• object-oriented modeling
• acceptance test generation