Efficacy of a Computer Tutor that Models Expert Human Tutors

2025-04-21

AI Summary

This study investigates how expert pedagogical knowledge influences tutoring effectiveness, specifically whether intelligent tutoring systems (ITSs) can match human expert tutors. Method: We developed a cognitively grounded ITS for high school biology that explicitly models expert teaching strategies—not merely domain knowledge—and conducted a 9-week randomized controlled trial to assess immediate and delayed learning outcomes. Contribution/Results: For the first time, we empirically validate that modeling expert pedagogy is a critical mechanism underlying high ITS efficacy. The system significantly outperformed the no-tutoring control group on both immediate posttest (d = 0.71, p < 0.001) and delayed posttest (d = 0.36, p < 0.01). Crucially, its long-term knowledge retention was statistically equivalent to that achieved by expert human tutors. Analyses employed logistic mixed-effects models to ensure robust inference across multiple time points.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Intelligent Robots: Behavior Learning & Control
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Tutoring is highly effective for promoting learning. However, the contribution of expertise to tutoring effectiveness is unclear and continues to be debated. We conducted a 9-week learning efficacy study of an intelligent tutoring system (ITS) for biology modeled on expert human tutors with two control conditions: human tutors who were experts in the domain but not in tutoring and a no-tutoring condition. All conditions were supplemental to classroom instruction, and students took learning tests immediately before and after tutoring sessions as well as delayed tests 1-2 weeks later. Analysis using logistic mixed-effects modeling indicates significant positive effects on the immediate post-test for the ITS (d =.71) and human tutors (d =.66) which are in the 99th percentile of meta-analytic effects, as well as significant positive effects on the delayed post-test for the ITS (d =.36) and human tutors (d =.39). We discuss implications for the role of expertise in tutoring and the design of future studies.

Problem

• Assess efficacy of expert-modeled computer tutor vs human tutors
• Compare learning outcomes with and without tutoring support
• Evaluate role of expertise in tutoring effectiveness

Innovation

• Intelligent tutoring system models expert human tutors
• Comparative study with human tutors and no tutoring
• Logistic mixed-effects modeling for efficacy analysis