Next Token Knowledge Tracing: Exploiting Pretrained LLM Representations to Decode Student Behaviour

2025-11-04

AI Summary

Existing knowledge tracing (KT) models often neglect the pedagogical semantics embedded in problem texts, limiting their predictive performance—particularly for cold-start students and novel problems. To address this, we propose LLM-KT, the first KT framework that reformulates knowledge tracing as a large language model (LLM)-driven next-token prediction task. LLM-KT jointly encodes student interaction sequences and raw problem text, leveraging the LLM’s contextualized semantic representations to dynamically model latent knowledge states. Crucially, it requires no parameter fine-tuning—instead, it directly harnesses the pre-trained LLM’s inherent linguistic and pedagogical understanding. Extensive experiments on multiple benchmark datasets demonstrate that LLM-KT significantly outperforms state-of-the-art neural KT models, achieving substantial gains in both prediction accuracy and generalization—especially under cold-start conditions. This work establishes a novel KT paradigm that unifies instructional content semantics with behavioral modeling, advancing the integration of domain-aware language understanding into educational AI.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Acquisition
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Modelling student knowledge is a key challenge when leveraging AI in education, with major implications for personalised learning. The Knowledge Tracing (KT) task aims to predict how students will respond to educational questions in learning environments, based on their prior interactions. Existing KT models typically use response correctness along with metadata like skill tags and timestamps, often overlooking the question text, which is an important source of pedagogical insight. This omission poses a lost opportunity while limiting predictive performance. We propose Next Token Knowledge Tracing (NTKT), a novel approach that reframes KT as a next-token prediction task using pretrained Large Language Models (LLMs). NTKT represents both student histories and question content as sequences of text, allowing LLMs to learn patterns in both behaviour and language. Our series of experiments significantly improves performance over state-of-the-art neural KT models and generalises much better to cold-start questions and users. These findings highlight the importance of question content in KT and demonstrate the benefits of leveraging pretrained representations of LLMs to model student learning more effectively.

Problem

• Predict student responses to educational questions using their interaction history
• Leverage question text content often overlooked by existing knowledge tracing models
• Improve generalization for cold-start scenarios with new questions and users

Innovation

• Reframes knowledge tracing as next-token prediction
• Represents student histories as text sequences
• Leverages pretrained LLMs to model learning patterns