Code Readability in the Age of Large Language Models: An Industrial Case Study from Atlassian

2025-01-20

AI Summary

This study investigates differences in code readability between large language model (LLM)-generated and human-written code, and their impact on developer trust and adoption intent. Method: Leveraging Atlassian’s proprietary LLM-based development agent, HULA, we conducted an industrial-scale survey, double-blind human annotation assessments, and controlled human-vs.-LLM code comparisons within authentic software development contexts. Contribution/Results: Our analysis provides the first systematic empirical validation that HULA-generated code exhibits readability on par with human-authored code. Results confirm that readability remains a critical determinant of developer trust and platform acceptance in LLM-augmented programming. These findings offer rigorous empirical support for the engineering deployment of LLM-generated code and directly enabled HULA’s enterprise-scale adoption across Atlassian.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Programmers spend a significant amount of time reading code during the software development process. This trend is amplified by the emergence of large language models (LLMs) that automatically generate code. However, little is known about the readability of the LLM-generated code and whether it is still important from practitioners' perspectives in this new era. In this paper, we conduct a survey to explore the practitioners' perspectives on code readability in the age of LLMs and investigate the readability of our LLM-based software development agents framework, HULA, by comparing its generated code with human-written code in real-world scenarios. Overall, the findings underscore that (1) readability remains a critical aspect of software development; (2) the readability of our LLM-generated code is comparable to human-written code, fostering the establishment of appropriate trust and driving the broad adoption of our LLM-powered software development platform.

Problem

• Code Readability
• Language Models
• Software Development

Innovation

• Large Model Code Readability
• Empirical Analysis
• Software Development Trust