Analyzing the Difficulty of Programming Assignments with Interpretable Knowledge Component Metrics

2026-07-03

AI Summary

This study addresses the challenge of interpreting the relationship between programming assignment difficulty and student performance, a gap exacerbated by the limited interpretability of existing predictive models that hinders effective pedagogical refinement. To bridge this gap, the authors propose an interpretable analytical framework grounded in knowledge components (KCs), which can be either expert-defined or automatically extracted using large language models (LLMs). By quantifying the number of KCs involved in each assignment and measuring the shift in KCs between consecutive tasks, the framework elucidates how assignment design influences learning outcomes. Empirical evaluation across three introductory programming course datasets reveals that assignments involving a greater number of KCs correlate with poorer student performance, and abrupt KC transitions are significantly associated with learning disruptions. These findings enable the identification of poorly designed assignments, offering actionable insights for instructional diagnosis and improvement.

Abstract

This research paper examines how Knowledge Components (KCs) - fine-grained concepts or skills required to solve programming tasks - can be used as interpretable signals for understanding assignment difficulty and student struggle in introductory programming courses. While prior work has focused on predictive models based on programming behavior, such models are often difficult to interpret and therefore hard to use for instructional decisions. We analyze KC-based metrics, including the number of KCs per assignment and changes in KC coverage between consecutive assignments. We examine correlations between the number of KCs and student performance on the assignment, and analyze changes in KCs across assignments to identify cases where performance declines without new concepts being introduced. Selected assignments are then qualitatively inspected to understand potential design issues. Our results on data from three introductory programming course datasets show that assignments involving more KCs are generally associated with lower performance, and that sudden shifts in required KCs can coincide with disruptions in learning progression. We also identify assignments where performance declines even though no new KCs are introduced, suggesting potential issues in task design or instruction, which could be examined with qualitative analysis. We propose an interpretable framework for analyzing programming assignments using KC-based metrics, with practical implications for instructors and course designers who want to better understand where and why students struggle, and how course materials might be improved. Our method can use KCs defined by an expert or extracted by an LLM, offering instructors an additional way to assess assignment quality beyond average correctness. It can be applied to any course with ordered assignments and measurable performance.

Problem

• Programming Assignments
• Difficulty Analysis
• Knowledge Components
• Student Struggle
• Interpretable Metrics

Innovation

• Knowledge Components
• interpretable metrics
• programming assignment difficulty
• learning progression
• instructional design