Prompt-with-Me: in-IDE Structured Prompt Management for LLM-Driven Software Engineering

2025-09-21

AI Summary

Large language models (LLMs) are increasingly deployed in software engineering, yet ad-hoc prompt management hinders reliability, reusability, and industrial adoption efficiency. Method: We propose the first software-engineering–specific four-dimensional prompt taxonomy—categorizing prompts by intent, role, software lifecycle phase, and prompt type—and design an integrated prompt management framework supporting sensitive information masking, automated template extraction, and language-optimized suggestions. Contribution/Results: Evaluated on 1,108 real-world prompts, our taxonomy achieves high classification accuracy. A user study with 11 practitioners demonstrates strong usability (System Usability Scale score = 73) and low cognitive load (NASA-TLX score = 21), significantly reducing redundant effort. This work establishes a scalable, integrable, and structured paradigm for LLM prompt engineering in software engineering contexts.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Software Engineering

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large Language Models are transforming software engineering, yet prompt management in practice remains ad hoc, hindering reliability, reuse, and integration into industrial workflows. We present Prompt-with-Me, a practical solution for structured prompt management embedded directly in the development environment. The system automatically classifies prompts using a four-dimensional taxonomy encompassing intent, author role, software development lifecycle stage, and prompt type. To enhance prompt reuse and quality, Prompt-with-Me suggests language refinements, masks sensitive information, and extracts reusable templates from a developer's prompt library. Our taxonomy study of 1108 real-world prompts demonstrates that modern LLMs can accurately classify software engineering prompts. Furthermore, our user study with 11 participants shows strong developer acceptance, with high usability (Mean SUS=73), low cognitive load (Mean NASA-TLX=21), and reported gains in prompt quality and efficiency through reduced repetitive effort. Lastly, we offer actionable insights for building the next generation of prompt management and maintenance tools for software engineering workflows.

Problem

• Ad hoc prompt management hinders reliability and reuse in LLM-driven software engineering
• Lack of structured prompt classification and integration into industrial development workflows
• Need for automated prompt refinement, sensitive information masking, and template extraction

Innovation

• Automatically classifies prompts using four-dimensional taxonomy
• Suggests language refinements and masks sensitive information
• Extracts reusable templates from developer prompt library