Evaluating a Data-Driven Redesign Process for Intelligent Tutoring Systems

2026-03-30

AI Summary

This study investigates the generalizability of data-driven approaches to reconstructing intelligent tutoring systems in non-preselected instructional units. Focusing on four middle school mathematics topics that had not undergone prior efficacy screening, the authors implemented a system redesign grounded in data-driven instructional design and learning behavior analysis, followed by a classroom-based randomized controlled trial to evaluate its impact. Although the intervention did not yield statistically significant gains in overall learning outcomes, it led to significant improvements in students’ engaged learning time, volume of skill practice, and breadth of knowledge acquisition. This work represents the first validation of data-driven reconstruction methods in uncurated instructional contexts, thereby extending their applicability to more authentic and diverse educational settings.

Abstract

Past research has defined a general process for the data-driven redesign of educational technologies and has shown that in carefully-selected instances, this process can help make systems more effective. In the current work, we test the generality of the approach by applying it to four units of a middle-school mathematics intelligent tutoring system that were selected not based on suitability for redesign, as in previous work, but on topic. We tested whether the redesigned system was more effective than the original in a classroom study with 123 students. Although the learning gains did not differ between the conditions, students who used the Redesigned Tutor had more productive time-on-task, a larger number of skills practiced, and greater total knowledge mastery. The findings highlight the promise of data-driven redesign even when applied to instructional units *not* selected as likely to yield improvement, as evidence of the generality and wide applicability of the method.

Problem

• data-driven redesign
• intelligent tutoring systems
• generality
• effectiveness
• classroom study

Innovation

• data-driven redesign
• intelligent tutoring systems
• classroom study
• generality
• productive time-on-task