Leveraging Large Language Models for Identifying Knowledge Components

2025-11-13

AI Summary

This study addresses the high cost of manual knowledge component (KC) annotation in adaptive learning systems and the redundancy issue prevalent in existing automated KC labeling methods under low-resource settings. We propose an LLM-based automated KC identification framework leveraging GPT-4o-mini. Methodologically, we introduce a hierarchical prompting strategy to enhance semantic fidelity of KCs and integrate a cosine-similarity-based embedding clustering mechanism for dynamic semantic merging and redundancy reduction. Evaluated on a dataset of 646 multiple-choice questions, our approach reduces the number of KCs from 569 to 428 while improving prediction accuracy (RMSE reduced to 0.4259), outperforming baseline automatic methods. To our knowledge, this is the first work to systematically incorporate semantic merging into an LLM-driven KC generation pipeline, balancing interpretability and practicality. The framework establishes an efficient, lightweight paradigm for KC modeling in small-scale educational datasets.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Knowledge Components (KCs) are foundational to adaptive learning systems, but their manual identification by domain experts is a significant bottleneck. While Large Language Models (LLMs) offer a promising avenue for automating this process, prior research has been limited to small datasets and has been shown to produce superfluous, redundant KC labels. This study addresses these limitations by first scaling a"simulated textbook"LLM prompting strategy (using GPT-4o-mini) to a larger dataset of 646 multiple-choice questions. We found that this initial automated approach performed significantly worse than an expert-designed KC model (RMSE 0.4285 vs. 0.4206) and generated an excessive number of KCs (569 vs. 101). To address the issue of redundancy, we proposed and evaluated a novel method for merging semantically similar KC labels based on their cosine similarity. This merging strategy significantly improved the model's performance; a model using a cosine similarity threshold of 0.8 achieved the best result, reducing the KC count to 428 and improving the RMSE to 0.4259. This demonstrates that while scaled LLM generation alone is insufficient, combining it with a semantic merging technique offers a viable path toward automating and refining KC identification.

Problem

• Automating knowledge component identification in adaptive learning systems
• Reducing redundant KC labels generated by large language models
• Improving model performance through semantic merging of similar KCs

Innovation

• Using GPT-4o-mini to generate knowledge components automatically
• Applying cosine similarity to merge redundant knowledge components
• Combining LLM generation with semantic merging for KC identification