Personalized Distractor Generation via MCTS-Guided Reasoning Reconstruction

2025-08-14

AI Summary

Existing methods struggle to accurately infer students’ specific reasoning errors from sparse response records, resulting in non-diagnostic distractors. Method: We introduce the first personalized distractor generation task and propose a training-free two-stage framework: (1) Monte Carlo Tree Search (MCTS) reconstructs individual reasoning paths by backtracking from a small set of incorrect answers; (2) a large language model (LLM) constructs student-specific misconception prototypes and guides personalized distractor generation for novel questions. Contribution/Results: Evaluated on data from 140 students, our approach significantly outperforms baselines. It achieves both strong population-level generalization and, for the first time, precise modeling of individual cognitive biases—enabling truly diagnostic, student-tailored distractors.

Technology Category

• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Distractors, incorrect but plausible answer choices in multiple-choice questions (MCQs), play a critical role in educational assessment by diagnosing student misconceptions. Recent work has leveraged large language models (LLMs) to generate shared, group-level distractors by learning common error patterns across large student populations. However, such distractors often fail to capture the diverse reasoning errors of individual students, limiting their diagnostic effectiveness. To address this limitation, we introduce the task of personalized distractor generation, which aims to generate tailored distractors based on individual misconceptions inferred from each student's past question-answering (QA) records, ensuring every student receives options that effectively exposes their specific reasoning errors. While promising, this task is challenging because each student typically has only a few QA records, which often lack the student's underlying reasoning processes, making training-based group-level approaches infeasible. To overcome this, we propose a training-free two-stage framework. In the first stage, we construct a student-specific misconception prototype by applying Monte Carlo Tree Search (MCTS) to recover the student's reasoning trajectories from past incorrect answers. In the second stage, this prototype guides the simulation of the student's reasoning on new questions, enabling the generation of personalized distractors that align with the student's recurring misconceptions. Experiments show that our approach achieves the best performance in generating plausible, personalized distractors for 140 students, and also effectively generalizes to group-level settings, highlighting its robustness and adaptability.

Problem

• Generating personalized distractors for individual student misconceptions
• Overcoming limited student QA records for reasoning reconstruction
• Using MCTS to simulate reasoning and generate tailored distractors

Innovation

• MCTS reconstructs reasoning from incorrect answers
• Training-free framework for personalized distractors
• Generates distractors aligning with student misconceptions