An Empirical Study on Low Code Programming using Traditional vs Large Language Model Support

2024-02-02

Journal of Systems and Software

AI Summary

Traditional low-code programming (LCP) and large language model (LLM)-enhanced LCP exhibit distinct technical foundations and practical implications, yet their comparative strengths, limitations, and lifecycle coverage—particularly during implementation—remain empirically underexplored. Method: Drawing on three years of Stack Overflow developer Q&A data, this study employs empirical analysis, topic modeling, and systematic comparison to characterize differences in coverage, applicability boundaries, and user pain points across the software lifecycle. Contribution/Results: We identify significant divergences between the two paradigms in end-to-end lifecycle support, adaptability, and inherent constraints—despite overlapping core use cases. Crucially, we propose a novel hybrid paradigm integrating visual programming languages with LLM-based agents. This work provides empirically grounded guidance for LCP tool evolution and offers actionable insights for developer technology selection.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Reasoning under Uncertainty: Probabilistic Programming

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Low-code programming (LCP) refers to programming using models at higher levels of abstraction, resulting in less manual and more efficient programming, and reduced learning effort for amateur developers. Many LCP tools have rapidly evolved and have benefited from the concepts of visual programming languages (VPLs) and programming by demonstration (PBD). With huge increase in interest in using large language models (LLMs) in software engineering, LLM-based LCP has began to become increasingly important. However, the technical principles and application scenarios of traditional approaches to LCP and LLM-based LCP are significantly different. Understanding these key differences and characteristics in the application of the two approaches to LCP by users is crucial for LCP providers in improving existing and developing new LCP tools, and in better assisting users in choosing the appropriate LCP technology. We conducted an empirical study of both traditional LCP and LLM-based LCP. We analyzed developers' discussions on Stack Overflow (SO) over the past three years and then explored the similarities and differences between traditional LCP and LLM-based LCP features and developer feedback. Our findings reveal that while traditional LCP and LLM-based LCP share common primary usage scenarios, they significantly differ in scope, limitations and usage throughout the software development lifecycle, particularly during the implementation phase. We also examine how LLMs impact and integrate with LCP, discussing the latest technological developments in LLM-based LCP, such as its integration with VPLs and the application of LLM Agents in software engineering.

Problem

• Compares traditional and LLM-based low-code programming approaches
• Analyzes developer discussions to identify similarities and differences
• Examines LLM integration impacts on low-code development lifecycle

Innovation

• Empirical study comparing traditional and LLM-based low-code programming
• Analyzing developer discussions on Stack Overflow for insights
• Exploring LLM integration with visual languages and agent applications