Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failure

2024-10-03

arXiv.org

AI Summary

Current large language models (LLMs) lack pedagogical strategy capabilities—such as delaying answer provision and orchestrating multi-turn instructional dialogues—in one-on-one mathematics tutoring. Method: This paper introduces *Pedagogical Steering*, the first framework to formalize teaching strategies as verifiable state-transition graphs, coupled with StratL, a novel prompt optimization algorithm enabling controllable guidance of LLM teaching behavior. Focusing on *Productive Failure* (PF), a well-established constructivist pedagogy, we implement a dialogue-based mathematics tutoring prototype. Contribution/Results: In a field study with 17 high school students in Singapore, our system successfully enforced strict adherence to the PF protocol by the LLM. We publicly release both the curated PF problem dataset and the implementation code. This work establishes a methodology for LLM-augmented personalized instruction that is interpretable, reproducible, and extensible.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

One-to-one tutoring is one of the most efficient methods of teaching. With the growing popularity of Large Language Models (LLMs), there have been efforts to create LLM based conversational tutors which can expand the benefits of one to one tutoring to everyone. However, current LLMs are trained primarily to be helpful assistants and lack crucial pedagogical skills. For example, they often quickly reveal the solution to the student and fail to plan for a richer multi turn pedagogical interaction. To use LLMs in pedagogical settings, they need to be steered to use effective teaching strategies: a problem we introduce as Pedagogical Steering. We develop StratL, an algorithm to optimize LLM prompts and steer it to follow a predefined multi-turn tutoring plan represented as a transition graph. As a case study, we create a prototype tutor for high school math following Productive Failure (PF), an advanced and effective learning design. To validate our approach in a real-world setting, we run a field study with 17 high school students in Singapore and show that StratL succeeds in steering the LLM to follow the PF tutoring strategy. Finally, we highlight challenges in Pedagogical Steering of LLMs and offer opportunities for further improvements by publishing a dataset of PF problems and our code.

Problem

• LLMs lack pedagogical skills for effective tutoring.
• Developed StratL to steer LLMs for multi-turn tutoring.
• Validated StratL with high school math in Singapore.

Innovation

• Developed StratL algorithm for LLM steering
• Implemented Productive Failure tutoring strategy
• Conducted field study with high school students