Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Models

2024-06-29

arXiv.org

AI Summary

This study addresses the lack of empirical investigation into systematic coding-style discrepancies between large language models (LLMs) and human developers. We first propose a taxonomy of coding-style inconsistency and conduct multi-model comparative experiments across three dimensions: readability, conciseness, and robustness. Our methodology integrates expert annotation, quantitative style assessment, and code-quality metric design. Key contributions include: (1) the first principled taxonomy of coding-style inconsistency; (2) empirical identification of prevalent LLM-specific stylistic deviations—including redundant comments, excessive defensive programming, and non-idiomatic syntax; and (3) experimental validation that such inconsistencies significantly impair code maintainability and team collaboration efficiency, alongside actionable mitigation strategies. The work provides both theoretical foundations and practical guidelines for enhancing the engineering viability of LLM-generated code.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) have brought a paradigm shift to the field of code generation, offering the potential to enhance the software development process. However, previous research mainly focuses on the accuracy of code generation, while coding style differences between LLMs and human developers remain under-explored. In this paper, we empirically analyze the differences in coding style between the code generated by mainstream Code LLMs and the code written by human developers, and summarize coding style inconsistency taxonomy. Specifically, we first summarize the types of coding style inconsistencies by manually analyzing a large number of generation results. We then compare the code generated by Code LLMs with the code written by human programmers in terms of readability, conciseness, and robustness. The results reveal that LLMs and developers have different coding styles. Additionally, we study the possible causes of these inconsistencies and provide some solutions to alleviate the problem.

Problem

• Investigates coding style differences between LLMs and human developers
• Analyzes readability, conciseness, and robustness in generated code
• Explores causes and solutions for coding style inconsistencies

Innovation

• Analyze coding style inconsistencies between LLMs and humans
• Compare code readability, conciseness, and robustness
• Provide solutions to reduce style inconsistencies