Prompt Adaptation as a Dynamic Complement in Generative AI Systems

2024-07-19

AI Summary

Under rapid generative AI model iteration, users’ ability to adapt to evolving models critically determines the translation of technological advancement into economic value. Method: This paper introduces *prompt adaptation*—users’ deliberate refinement of input prompts—as a dynamic complementarity mechanism in generative AI. We systematically quantify its impact via online controlled experiments, analysis of over 18,000 real-world human-AI interactions, and image similarity-based evaluation. Contribution/Results: Approximately 49% of DALL·E 3’s performance gain stems from user-initiated prompt adjustments; replacing such human adaptation with automated prompt rewriting incurs a 58% loss in upgrade benefits. These findings demonstrate that user adaptive behavior accounts for nearly half of the value generated by model upgrades, underscoring its pivotal role in human-AI co-creation. The study provides empirical grounding for AI deployment strategies and human-centered interface design.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: Prompt Engineering / Prompting
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

As generative AI systems rapidly improve, a key question emerges: How do users keep up-and what happens if they fail to do so. Drawing on theories of dynamic capabilities and IT complements, we examine prompt adaptation-the adjustments users make to their inputs in response to evolving model behavior-as a mechanism that helps determine whether technical advances translate into realized economic value. In a preregistered online experiment with 1,893 participants, who submitted over 18,000 prompts and generated more than 300,000 images, users attempted to replicate a target image in 10 tries using one of three randomly assigned models: DALL-E 2, DALL-E 3, or DALL-E 3 with automated prompt rewriting. We find that users with access to DALL-E 3 achieved higher image similarity than those with DALL-E 2-but only about half of this gain (51%) came from the model itself. The other half (49%) resulted from users adapting their prompts in response to the model's capabilities. This adaptation emerged across the skill distribution, was driven by trial-and-error, and could not be replicated by automated prompt rewriting, which erased 58% of the performance improvement associated with DALL-E 3. Our findings position prompt adaptation as a dynamic complement to generative AI-and suggest that without it, a substantial share of the economic value created when models advance may go unrealized.

Problem

• How users adapt prompts to evolving AI model capabilities
• Measuring economic value from user-AI interaction improvements
• Comparing manual vs automated prompt adaptation effectiveness

Innovation

• Prompt adaptation enhances generative AI performance
• User trial-and-error drives effective prompt adjustments
• Automated prompt rewriting reduces performance gains