Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts

2024-09-19

arXiv.org

AI Summary

This paper investigates how developers build generative AI software incorporating natural language prompts, focusing on “prompt programming”—an emerging paradigm distinct from traditional programming in its foundational principles. Method: Applying Straussian grounded theory, we conducted in-depth interviews with 20 cross-domain developers to systematically characterize prompt programming as a novel programming phenomenon. Contribution/Results: We formally define prompt programming as centered on constructing a *mental model of foundation model behavior*, rather than a logical model of code execution. The study identifies 15 key practice characteristics—such as high iteration frequency, inherent ambiguity, and dependence on opaque model internals—revealing fundamental regularities unique to prompt development. These findings provide empirical grounding and conceptual scaffolding for designing prompt-specific tools, debugging methodologies, evaluation frameworks, and advancing software engineering theory.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Reasoning under Uncertainty: Probabilistic Programming
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Generative pre-trained models power intelligent software features used by millions of users controlled by developer-written natural language prompts. Despite the impact of prompt-powered software, little is known about its development process and its relationship to programming. In this work, we argue that some prompts are programs and that the development of prompts is a distinct phenomenon in programming known as"prompt programming". We develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development across a variety of contexts, models, domains, and prompt structures. We contribute 15 observations to form a preliminary understanding of current prompt programming practices. For example, rather than building mental models of code, prompt programmers develop mental models of the foundation model (FM)'s behavior on the prompt by interacting with the FM. While prior research shows that experts have well-formed mental models, we find that prompt programmers who have developed dozens of prompts still struggle to develop reliable mental models. Our observations show that prompt programming differs from traditional software development, motivating the creation of prompt programming tools and providing implications for software engineering stakeholders.

Problem

• Understanding prompt programming as a distinct development practice
• Exploring mental models in prompt programming versus traditional coding
• Identifying challenges in developing reliable prompts for generative models

Innovation

• Prompts as programs in software development
• Grounded theory study on prompt programming
• Mental models of foundation model behavior