Tracing Mathematical Proficiency Through Problem-Solving Processes

2025-11-28

AI Summary

Traditional knowledge tracing (KT) methods rely solely on binary response correctness, failing to capture the latent structural composition of students’ mathematical competencies and suffering from severe interpretability limitations. Method: We propose KT-PSP, the first problem-solving process-driven KT framework. It employs a teacher–student–teacher large language model pipeline to automatically extract multidimensional mathematical proficiency signals from students’ step-by-step solution traces. We accompany this with KT-PSP-25—a newly released dataset covering 25 fine-grained mathematical competency dimensions and annotated solution processes. Our approach integrates procedural sequence modeling, task-adaptive proficiency metric design, and an intrinsic interpretability evaluation mechanism. Contribution/Results: Experiments demonstrate that KT-PSP significantly outperforms state-of-the-art baselines in prediction accuracy. Crucially, it enables dimension-aware knowledge state attribution and diagnostic feedback, establishing a novel, empirically grounded paradigm for interpretable KT.

Technology Category

• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Knowledge Tracing (KT) aims to model student's knowledge state and predict future performance to enable personalized learning in Intelligent Tutoring Systems. However, traditional KT methods face fundamental limitations in explainability, as they rely solely on the response correctness, neglecting the rich information embedded in students' problem-solving processes. To address this gap, we propose Knowledge Tracing Leveraging Problem-Solving Process (KT-PSP), which incorporates students' problem-solving processes to capture the multidimensional aspects of mathematical proficiency. We also introduce KT-PSP-25, a new dataset specifically designed for the KT-PSP. Building on this, we present StatusKT, a KT framework that employs a teacher-student-teacher three-stage LLM pipeline to extract students' MP as intermediate signals. In this pipeline, the teacher LLM first extracts problem-specific proficiency indicators, then a student LLM generates responses based on the student's solution process, and a teacher LLM evaluates these responses to determine mastery of each indicator. The experimental results on KT-PSP-25 demonstrate that StatusKT improves the prediction performance of existing KT methods. Moreover, StatusKT provides interpretable explanations for its predictions by explicitly modeling students' mathematical proficiency.

Problem

• Model student knowledge using problem-solving processes
• Improve explainability of knowledge tracing methods
• Predict student performance with interpretable proficiency indicators

Innovation

• Incorporates problem-solving processes for multidimensional proficiency
• Uses teacher-student-teacher LLM pipeline to extract proficiency indicators
• Provides interpretable explanations by explicitly modeling mathematical proficiency