🤖 AI Summary
Existing LLM code-generation evaluations rely heavily on controlled benchmarks (e.g., HumanEval), which poorly reflect real-world development practices. Method: This paper presents the first large-scale empirical study of code generated by ChatGPT and GitHub Copilot on GitHub, integrating repository crawling, language identification, commit-history tracing, complexity measurement, and pattern mining to characterize distribution, evolution, and maintenance properties. Contributions/Results: (1) Quantifies low real-world adoption: LLM-generated code constitutes <1.2% of total codebase volume on average; only 3–8% of such code undergoes modification for bug fixes; and generation is heavily skewed toward Python, Java, and TypeScript. (2) Reveals that associated projects tend to be small-scale, actively evolving, yet critically deficient in documentation. (3) Bridges the gap between controlled benchmarking and engineering practice, establishing an empirical foundation for assessing LLM code trustworthiness and guiding tool optimization.
📝 Abstract
The increasing use of Large Language Models (LLMs) in software development has garnered significant attention from researchers assessing the quality of the code they generate. However, much of the research focuses on controlled datasets such as HumanEval, which fail to adequately represent how developers actually utilize LLMs' code generation capabilities or clarify the characteristics of LLM-generated code in real-world development scenarios. To bridge this gap, our study investigates the characteristics of LLM-generated code and its corresponding projects hosted on GitHub. Our findings reveal several key insights: (1) ChatGPT and Copilot are the most frequently utilized for generating code on GitHub. In contrast, there is very little code generated by other LLMs on GitHub. (2) Projects containing ChatGPT/Copilot-generated code are often small and less known, led by individuals or small teams. Despite this, most projects are continuously evolving and improving. (3) ChatGPT/Copilot is mainly utilized for generating Python, Java, and TypeScript scripts for data processing and transformation. C/C++ and JavaScript code generation focuses on algorithm and data structure implementation and user interface code. Most ChatGPT/Copilot-generated code snippets are relatively short and exhibit low complexity. (4) Compared to human-written code, ChatGPT/Copilot-generated code exists in a small proportion of projects and generally undergoes fewer modifications. Additionally, modifications due to bugs are even fewer, ranging from just 3% to 8% across different languages. (5) Most comments on ChatGPT/Copilot-generated code lack detailed information, often only stating the code's origin without mentioning prompts, human modifications, or testing status. Based on these findings, we discuss the implications for researchers and practitioners.