Investigating the Robustness of Knowledge Tracing Models in the Presence of Student Concept Drift

2025-11-01

AI Summary

This study investigates the robustness of knowledge tracing (KT) models under student concept drift, examining performance degradation during long-term cross-academic-year deployment. Leveraging five years of real-world online learning data, we conduct a longitudinal empirical evaluation of the Bayesian Knowledge Tracing (BKT) model and three representative attention-based KT models. Results demonstrate that all models suffer significant accuracy decline due to concept drift; however, BKT exhibits the highest stability, whereas more complex models—particularly Transformer-based architectures—experience accelerated predictive deterioration over time. Crucially, this work provides the first quantitative evidence of a negative correlation between KT model robustness and architectural complexity, offering key empirical insights for the sustainable deployment of educational AI systems. To ensure reproducibility and foster further research, all datasets and source code are publicly released.

Technology Category

• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Knowledge Representation and Reasoning: Knowledge Acquisition
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Knowledge Tracing (KT) has been an established problem in the educational data mining field for decades, and it is commonly assumed that the underlying learning process be- ing modeled remains static. Given the ever-changing land- scape of online learning platforms (OLPs), we investigate how concept drift and changing student populations can im- pact student behavior within an OLP through testing model performance both within a single academic year and across multiple academic years. Four well-studied KT models were applied to five academic years of data to assess how suscep- tible KT models are to concept drift. Through our analysis, we find that all four families of KT models can exhibit de- graded performance, Bayesian Knowledge Tracing (BKT) remains the most stable KT model when applied to newer data, while more complex, attention based models lose pre- dictive power significantly faster. To foster more longitu- dinal evaluations of KT models, the data used to conduct our analysis is available at https://osf.io/hvfn9/?view_ only=b936c63dfdae4b0b987a2f0d4038f72a

Problem

• Investigating robustness of Knowledge Tracing models
• Analyzing impact of concept drift on student behavior
• Evaluating model performance degradation over time

Innovation

• Evaluated knowledge tracing models across multiple academic years
• Identified Bayesian Knowledge Tracing as most stable model
• Found attention-based models degrade faster with concept drift