Prompt Coach: An Empirical Evaluation of an Agentic Tutor for Learning Prompt Engineering in Software Development

2026-07-07

AI Summary

This work addresses the challenge that software developers face in effectively learning prompt engineering, a skill whose dynamic, interactive, and context-dependent nature is poorly served by traditional instructional methods. To bridge this gap, the paper introduces Prompt Coach—the first system that combines agent-based tutoring with Socratic questioning embedded directly within the IDE. Prompt Coach delivers context-aware, personalized guidance by analyzing the developer’s current codebase and large language model (LLM) interactions to evaluate prompt quality across multiple dimensions and support iterative refinement. The system integrates LLM behavior analysis, multidimensional evaluation metrics, IDE integration, and an interactive agent mechanism. In a 60-minute study with 15 professional developers, participants demonstrated significant improvements in prompt quality—particularly in commonly overlooked dimensions—and consistently reported enhanced prompt-writing capabilities.

Abstract

Prompt engineering has emerged as a critical yet undertaught skill for software developers, one that traditional learning approaches are ill-equipped to support given its evolving, interactive, and context-dependent nature. In this paper, we introduce Prompt Coach (PC), an agentic tutor that helps developers learn how to craft high-quality code-generation prompts through Socratic guidance embedded in-flow within their IDE. PC evaluates prompt quality across multiple dimensions and surfaces targeted questions to guide self-correction, grounded in the developer's codebase and the behavior of the target LLM. We present an early empirical study with 15 professional developers combining quantitative prompt quality scoring with qualitative perception measures. Participants showed statistically significant improvements after a single 60-minute session, with the largest gains across dimensions commonly overlooked by developers. They also reported strong trust, high adoption readiness, and unanimous agreement that PC improved their prompt-writing skills.

Problem

• prompt engineering
• software development
• agentic tutor
• learning support
• interactive learning

Innovation

• agentic tutor
• prompt engineering
• Socratic guidance
• in-flow learning
• IDE-integrated AI