Using LLMs for Knowledge Component-level Correctness Labeling in Open-ended Coding Problems

2026-02-19

AI Summary

This study addresses the scarcity of fine-grained, knowledge component (KC)-level correctness labels in open-ended coding problems, which limits the accuracy of student modeling and learning curve estimation. To overcome the limitations of traditional problem-level label propagation, this work proposes a novel temporal context-aware Code-KC dynamic mapping mechanism that leverages large language models (LLMs) to automatically annotate KC-level correctness from student-submitted code. Integrating knowledge tracing, the Additive Factors Model, and power-law learning curve modeling, the generated KC labels significantly enhance the alignment between empirical learning curves and cognitive theory, outperform baseline methods in predictive performance, and demonstrate strong agreement with expert annotations.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Fine-grained skill representations, commonly referred to as knowledge components (KCs), are fundamental to many approaches in student modeling and learning analytics. However, KC-level correctness labels are rarely available in real-world datasets, especially for open-ended programming tasks where solutions typically involve multiple KCs simultaneously. Simply propagating problem-level correctness to all associated KCs obscures partial mastery and often leads to poorly fitted learning curves. To address this challenge, we propose an automated framework that leverages large language models (LLMs) to label KC-level correctness directly from student-written code. Our method assesses whether each KC is correctly applied and further introduces a temporal context-aware Code-KC mapping mechanism to better align KCs with individual student code. We evaluate the resulting KC-level correctness labels in terms of learning curve fit and predictive performance using the power law of practice and the Additive Factors Model. Experimental results show that our framework leads to learning curves that are more consistent with cognitive theory and improves predictive performance, compared to baselines. Human evaluation further demonstrates substantial agreement between LLM and expert annotations.

Problem

• Knowledge Component
• Correctness Labeling
• Open-ended Coding Problems
• Student Modeling
• Learning Analytics

Innovation

• Large Language Models
• Knowledge Component Labeling
• Open-ended Coding Problems
• Temporal Context-aware Mapping
• Learning Curve Modeling