What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

2026-09-11

AI Summary

研究对比了人类编写与AI生成代码的质量差异，通过分析结构复杂性、自然性和静态分析结果，发现AI代码更简洁但存在不同类型的缺陷和安全问题。

Abstract

AI coding assistants are becoming co-authors of production software, yet their evaluation centers on functional correctness, leaving open whether their code differs from human code in the quality dimensions dominating lifecycle cost. We compare human-written and AI-generated code at scale: 787,562 function pairs across Python, Java, and C, each human function mined from open-source repositories paired with implementations generated from its docstring by three AI assistants (OpenAI GPT models, DeepSeek-Coder, Qwen2.5-Coder). We characterize structural complexity and statistical naturalness, and map static-analysis findings onto Orthogonal Defect Classification for defects and the Common Weakness Enumeration for vulnerabilities, making authors and languages directly comparable. AI-generated code is structurally compressed and stylistically templated: roughly half the size and branching of human code, clustering apart at the style level. Defect profiles differ in kind: human code concentrates issues of mature codebases, AI code repetitive boilerplate; security is language-dependent, with LLMs producing more, and more severe, findings in Python and Java but fewer high-severity memory-safety findings than humans in C. Once size is controlled for, complexity metrics carry little signal, while naturalness separates authors. Finally, we release CQBench, a benchmark of 27,346 issue-prone tasks with baselines and an evaluation pipeline for quality assurance and security testing.

Problem

• AI-generated code
• human-written code
• quality dimensions
• software lifecycle cost
• static analysis

Innovation

• structural complexity
• naturalness
• static-analysis
• security vulnerabilities
• CQBench