Cold Start Problem: An Experimental Study of Knowledge Tracing Models with New Students

2025-05-22

AI Summary

This work addresses the *new-student cold-start problem* in knowledge tracing (KT)—i.e., predicting the knowledge states of previously unseen learners from zero or few interactions. We introduce, for the first time, a rigorous *student-level train-test split* paradigm and establish a dedicated zero-/few-shot generalization benchmark for new learners. Using three representative KT models—DKT, DKVMN, and SAKT—we conduct systematic evaluation on the ASSISTments 2009, 2015, and 2017 datasets. Results show that all models suffer substantial initial accuracy degradation under zero-shot conditions, with gradual recovery as interaction count increases; SAKT achieves the best performance yet still exhibits pronounced generalization bottlenecks. Our study empirically exposes a fundamental limitation of current KT models in cross-student generalization, providing both an evidence-based foundation and a standardized evaluation protocol to guide the development of KT methods with robust generalization capabilities.

Technology Category

• Knowledge Representation and Reasoning: Knowledge Acquisition
• Machine Learning: Kernel Methods
• Data Mining & Knowledge Management: Knowledge Acquisition from the Web

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

KnowledgeTracing (KT) involves predicting students' knowledge states based on their interactions with Intelligent Tutoring Systems (ITS). A key challenge is the cold start problem, accurately predicting knowledge for new students with minimal interaction data. Unlike prior work, which typically trains KT models on initial interactions of all students and tests on their subsequent interactions, our approach trains models solely using historical data from past students, evaluating their performance exclusively on entirely new students. We investigate cold start effects across three KT models: Deep Knowledge Tracing (DKT), Dynamic Key-Value Memory Networks (DKVMN), and Self-Attentive Knowledge Tracing (SAKT), using ASSISTments 2009, 2015, and 2017 datasets. Results indicate all models initially struggle under cold start conditions but progressively improve with more interactions; SAKT shows higher initial accuracy yet still faces limitations. These findings highlight the need for KT models that effectively generalize to new learners, emphasizing the importance of developing models robust in few-shot and zero-shot learning scenarios

Problem

• Addressing cold start problem in Knowledge Tracing for new students
• Evaluating KT models' performance with minimal interaction data
• Improving generalization of KT models for few-shot learning

Innovation

• Trains models using only historical student data
• Evaluates performance on entirely new students
• Tests three KT models under cold start