MalruleLib: Large-Scale Executable Misconception Reasoning with Step Traces for Modeling Student Thinking in Mathematics

2026-01-06

arXiv.org

AI Summary

Students often exhibit systematic mathematical errors due to repeatedly applying self-consistent but incorrect problem-solving rules, known as malrules, which are challenging to diagnose across diverse problem contexts. This work proposes MalruleLib, a framework that formalizes 101 empirically documented mathematical misconceptions from educational research into an executable, parameterized rule library for the first time. Integrated with 67 problem templates derived from prior studies, the framework generates over one million step-by-step solution trajectories encompassing both correct and erroneous reasoning paths. MalruleLib enables cross-template malrule inference and controllable evaluation, offering a scalable infrastructure for student cognitive modeling. Experiments on nine large language models reveal that direct cross-template malrule prediction accuracy is 26% lower than baseline performance, while incorporating student step trajectories improves accuracy by 3–15%, highlighting current models’ limitations in generalizing erroneous reasoning patterns.

Technology Category

• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Student mistakes in mathematics are often systematic: a learner applies a coherent but wrong procedure and repeats it across contexts. We introduce MalruleLib, a learning-science-grounded framework that translates documented misconceptions into executable procedures, drawing on 67 learning-science and mathematics education sources, and generates step-by-step traces of malrule-consistent student work. We formalize a core student-modeling problem as Malrule Reasoning Accuracy (MRA): infer a misconception from one worked mistake and predict the student's next answer under cross-template rephrasing. Across nine language models (4B-120B), accuracy drops from 66% on direct problem solving to 40% on cross-template misconception prediction. MalruleLib encodes 101 malrules over 498 parameterized problem templates and produces paired dual-path traces for both correct reasoning and malrule-consistent student reasoning. Because malrules are executable and templates are parameterizable, MalruleLib can generate over one million instances, enabling scalable supervision and controlled evaluation. Using MalruleLib, we observe cross-template degradations of 10-21%, while providing student step traces improves prediction by 3-15%. We release MalruleLib as infrastructure for educational AI that models student procedures across contexts, enabling diagnosis and feedback that targets the underlying misconception.

Problem

• misconception
• malrule
• student modeling
• mathematical reasoning
• cross-template prediction

Innovation

• executable misconceptions
• malrule reasoning
• step-by-step traces
• cross-template generalization
• student modeling