An Exploratory Study on LLM-Generated Code and Comments in Code Repositories

2026-07-02

AI Summary

This study addresses growing industry concerns about the practicality and naturalness of code generated by large language models (LLMs) by systematically examining the usage patterns and defect associations of LLM-generated code and comments in active enterprise and community repositories from 2021 to 2025. For the first time, it contrasts the distribution of LLM-generated content between these two repository types through an empirical analysis integrating multiple detection tools, code clone detection, syntactic quality assessment, and manually labeled defect data. The findings reveal that the proportion of LLM-generated code has steadily declined over time and is predominantly confined to test cases, while comment generation remains stable yet exhibits low syntactic correctness. Enterprise repositories incorporate more LLM-generated content overall, which shows virtually no direct association with known defects, suggesting that such content is characterized by low risk but high functional limitations in real-world practice.

Abstract

The use of LLMs in software development has become increasingly widespread on tasks such as code generation and summarization. Reports from large technology companies showed that around 20% to 30% of their code are generated by LLMs. However, there remains skepticism about the practical usage of LLM-generated code and comments, such as concerns on more time for debugging the generated code and the unnaturalness of the generated comments. In this paper, we study the code and comments detected as likely to be generated by LLMs and their characteristics, the differences between company- and community-maintained repositories, and how likely bugs are associated with LLM-generated code. We conduct extensive experiments on active company- and community-maintained repositories from 2021 to 2025 using various tools and techniques that detect code and comments generated by LLMs. Based on our detector-based proxy analysis, the results suggest that code detected as likely to be generated by LLMs decreased over time and appeared frequently in test cases, while that of comments remains relatively stable. Proxy results further suggest that code detected as likely to be generated by LLMs shows substantial intra-repository code clones, whereas comments exhibit a relatively low proportion of grammatically correct sentences. In addition, the company-maintained repositories show a higher percentage of code and comments detected as likely to be generated by LLMs, and only a small percentage of the human-labelled bugs are detected as being likely associated with LLM-generated code.

Problem

• LLM-generated code
• code comments
• software repositories
• code quality
• bug association

Innovation

• LLM-generated code
• detector-based proxy analysis
• code comments quality
• intra-repository clones
• empirical software engineering