Using LLMs in Software Design: An Empirical Study of GitHub and A Practitioner Survey

2026-05-02

AI Summary

This study addresses the lack of systematic understanding regarding the practical applications, benefits, and limitations of large language models (LLMs) in software design. Employing a mixed-methods approach, it analyzes 291 developer–ChatGPT interaction logs from GitHub alongside survey responses from 65 practitioners, combining qualitative content analysis with quantitative statistics. The research empirically identifies nine usage scenarios, seven core advantages, and six key limitations of LLMs in software design from both open-source practice and practitioner perspectives. Findings indicate that developers primarily leverage LLMs during detailed design to acquire knowledge and generate design-related code, deriving significant value in architecture, data modeling, and design patterns. However, challenges persist, including verbose outputs, non-executable code, and hallucinations stemming from context dependency, offering empirical grounding for future tool development.

Abstract

Recent advancements in Large Language Models (LLMs) have demonstrated significant potential across a wide range of software engineering tasks, including software design, an area traditionally regarded as highly dependent on human expertise and judgment. However, there has been little research focusing on how LLMs are used in software design, nor on the associated benefits and drawbacks. This paper aims to bridge this gap by empirically investigating how software developers utilize LLMs in the context of software design. We conduct a mixed-methods study, combining a mining study of 291 developer-ChatGPT conversations shared on GitHub with a survey of 65 software practitioners. Our findings reveal nine distinct categories of design tasks supported by ChatGPT, including architecture design, data model design, and the use of design patterns. We further characterize developer-ChatGPT interactions, showing that developers primarily use ChatGPT for knowledge acquisition and design-related code generation, with most tasks situated at the detailed design level. The study identifies seven key benefits of utilizing LLMs in software design as perceived by developers, such as better technology selection and the early detection of design flaws. We also uncover six limitations, including the generation of overly lengthy and difficult-to-read outputs, the creation of inexecutable or incorrect code, and a heavy reliance on context that can lead to hallucinated results. These findings provide an evidence-based characterization of current LLM use in software design from both open-source and practitioner perspectives, highlighting a tension between perceived benefits and limitations, which lays a foundation for future research and the development of effective techniques and tools to integrate LLMs into software design practices.

Problem

• Large Language Models
• Software Design
• Empirical Study
• Developer Practices
• ChatGPT

Innovation

• Large Language Models
• Software Design
• Empirical Study
• Developer Survey
• ChatGPT