PromptDecipher: Supporting AI Tutor Authoring Through Editable Simulated Interactions

2026-05-15

AI Summary

This work addresses the challenge educators face in developing AI tutoring chatbots due to limited capacity for systematic testing, which often compromises learning design, interaction design, and quality assurance. To bridge this gap, the paper proposes a quality-centric redesign of the authoring workflow: instructors directly correct undesirable responses in simulated dialogues, and the system automatically analyzes these corrections to generate targeted prompt rewrites, subsequently validating their effectiveness in predefined test scenarios. By replacing abstract prompt engineering with interactive revision, and integrating large language model–driven dialogue simulation, automated prompt rewriting, and a human-in-the-loop editing interface, the approach substantially lowers the technical barrier for non-technical educators. The system has been deployed in the “AI for Educators” course, serving hundreds of university instructors, and includes open-sourced code and an interactive prototype.

Abstract

Chatbots have long been explored as tools to support learning, and recent advances in large language models have significantly expanded the availability of platforms for educators to author AI tutoring chatbots. Yet effective authorship demands more than writing a system prompt; it requires educators to act as learning designers, AI interaction designers, and QA engineers. In practice, however, teachers rarely fulfill these roles. Our formative study found that virtually none systematically tested their bots before deploying them to students. To address this gap, we present PromptDecipher, a system that restructures the authoring workflow around a direct correction-based interaction rather than writing abstract system prompts, teachers interact with a live chat preview and edit undesirable bot responses. An automated pipeline then analyzes the correction, proposes a targeted system prompt rewrite, and validates the change across pre-defined test scenarios. This enforces QA as a first-class activity and scaffolds teachers in roles they would otherwise skip. PromptDecipher will be deployed in an AI for Educators course enrolling hundreds of higher-education instructors. A live prototype (https://teacher-prompting.vercel.app/), an anonymized codebase (https://anonymous.4open.science/r/teacher-prompting-2EDF/), and anonymized demo (https://tinyurl.com/las-prompt-decipher-demo) are available via links in the footnote.

Problem

• AI tutoring
• prompt authoring
• teacher support
• quality assurance
• chatbot design

Innovation

• prompt engineering
• AI tutoring
• interactive authoring
• quality assurance
• large language models