PromptPilot: Improving Human-AI Collaboration Through LLM-Enhanced Prompt Engineering

2025-10-01

AI Summary

In knowledge-intensive tasks, non-expert users struggle to craft effective prompts, limiting the practical efficacy of large language models (LLMs). To address this, we introduce PromptGuide—the first interactive, human-in-the-loop prompt engineering system, grounded in four empirically derived design principles. PromptGuide integrates LLM-based reasoning with principled human-computer interaction techniques to form a prompt enhancement framework that provides real-time feedback, strategy recommendations, and closed-loop optimization of user intent. A double-blind randomized controlled study (N=80) demonstrates that users employing PromptGuide achieve a significantly higher median task accuracy (78.3 vs. 61.7 for controls; p<0.01), alongside substantial improvements in efficiency, usability, and operational autonomy (all p<0.01). This work establishes a validated, interactive paradigm for prompt engineering that lowers the barrier to LLM adoption and strengthens user agency.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Humans and AI: Intelligent User Interfaces
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Effective prompt engineering is critical to realizing the promised productivity gains of large language models (LLMs) in knowledge-intensive tasks. Yet, many users struggle to craft prompts that yield high-quality outputs, limiting the practical benefits of LLMs. Existing approaches, such as prompt handbooks or automated optimization pipelines, either require substantial effort, expert knowledge, or lack interactive guidance. To address this gap, we design and evaluate PromptPilot, an interactive prompting assistant grounded in four empirically derived design objectives for LLM-enhanced prompt engineering. We conducted a randomized controlled experiment with 80 participants completing three realistic, work-related writing tasks. Participants supported by PromptPilot achieved significantly higher performance (median: 78.3 vs. 61.7; p = .045, d = 0.56), and reported enhanced efficiency, ease-of-use, and autonomy during interaction. These findings empirically validate the effectiveness of our proposed design objectives, establishing LLM-enhanced prompt engineering as a viable technique for improving human-AI collaboration.

Problem

• Users struggle to craft effective prompts for LLMs
• Existing prompt engineering methods lack interactive guidance
• Need to improve human-AI collaboration through enhanced prompting

Innovation

• Interactive prompting assistant for human-AI collaboration
• Empirically derived design objectives for LLM enhancement
• Randomized controlled experiment validating performance improvement