Prototyping with Prompts: Emerging Approaches and Challenges in Generative AI Design for Collaborative Software Teams

2024-02-27

AI Summary

This paper addresses the challenge of operationalizing generative AI within collaborative software engineering teams. Drawing on a design study with 39 industry experts—including field observations, semi-structured interviews, and multi-role workshops—we systematically investigate how prompt engineering supports cross-functional AI prototyping and iterative co-design. Our study is the first to characterize three core phenomena in collaborative prompt prototyping: (1) the emergent construction of shared coordination norms, (2) dynamic role evolution across developers, domain experts, and AI specialists, and (3) context-sensitive evaluation mechanisms for prompt efficacy. We propose a generative-content-feature-driven rapid iteration paradigm and distill a reusable prompt prototyping strategy framework. Key technical challenges—including model opacity and example overfitting—are empirically identified. The findings provide both methodological grounding and actionable practice guidelines for industrial software teams, advancing the shift from generative AI as a technical capability to a collaborative design enabler.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Multiagent Systems: Teamwork

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Generative AI models are increasingly being integrated into human task workflows, enabling the production of expressive content across a wide range of contexts. Unlike traditional human-AI design methods, the new approach to designing generative capabilities focuses heavily on prompt engineering strategies. This shift requires a deeper understanding of how collaborative software teams establish and apply design guidelines, iteratively prototype prompts, and evaluate them to achieve specific outcomes. To explore these dynamics, we conducted design studies with 39 industry professionals, including UX designers, AI engineers, and product managers. Our findings highlight emerging practices and role shifts in AI system prototyping among multistakeholder teams. We observe various prompting and prototyping strategies, highlighting the pivotal role of to-be-generated content characteristics in enabling rapid, iterative prototyping with generative AI. By identifying associated challenges, such as the limited model interpretability and overfitting the design to specific example content, we outline considerations for generative AI prototyping.

Problem

• Exploring prompt engineering in generative AI design
• Understanding collaborative team dynamics in AI prototyping
• Addressing challenges like model interpretability in prototyping

Innovation

• Focus on prompt engineering strategies
• Iterative prototyping with generative AI
• Addressing model interpretability challenges