Exploring Conversational Design Choices in LLMs for Pedagogical Purposes: Socratic and Narrative Approaches for Improving Instructor's Teaching Practice

2025-09-15

AI Summary

Existing large language model (LLM)–based educational tools struggle with heterogeneous teacher AI attitudes and teaching experience, hindering effective adaptation to pedagogical contexts. Method: We propose TeaPT, a dual-path dialogue design framework integrating Socratic questioning (to foster reflective thinking) and narrative-based suggestions (to support experiential externalization), grounded in mixed-methods empirical evaluation with 41 higher education instructors. Contribution/Results: Results show the Socratic path significantly enhances overall engagement, while the narrative path is preferentially adopted by experienced teachers—revealing a moderating effect of teaching experience. Teacher AI attitude also significantly predicts interaction preferences. This work is the first to systematically embed two foundational educational dialogue paradigms into LLM dialogue architecture, empirically uncovering how teacher characteristics moderate LLM interaction preferences. It advances a theoretically grounded, interpretable framework for personalized, pedagogically responsive AI support in teaching.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Interaction Techniques and Devices
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) typically generate direct answers, yet they are increasingly used as learning tools. Studying instructors' usage is critical, given their role in teaching and guiding AI adoption in education. We designed and evaluated TeaPT, an LLM for pedagogical purposes that supports instructors' professional development through two conversational approaches: a Socratic approach that uses guided questioning to foster reflection, and a Narrative approach that offers elaborated suggestions to extend externalized cognition. In a mixed-method study with 41 higher-education instructors, the Socratic version elicited greater engagement, while the Narrative version was preferred for actionable guidance. Subgroup analyses further revealed that less-experienced, AI-optimistic instructors favored the Socratic version, whereas more-experienced, AI-cautious instructors preferred the Narrative version. We contribute design implications for LLMs for pedagogical purposes, showing how adaptive conversational approaches can support instructors with varied profiles while highlighting how AI attitudes and experience shape interaction and learning.

Problem

• Exploring conversational design choices in LLMs for pedagogical purposes
• Evaluating Socratic and Narrative approaches to improve teaching practice
• Addressing how AI attitudes and experience shape instructor interaction

Innovation

• Socratic questioning for reflection
• Narrative suggestions for cognition
• Adaptive conversational approaches design