Tell Me Who Your Students Are: GPT Can Generate Valid Multiple-Choice Questions When Students' (Mis)Understanding Is Hinted

2025-05-09

AI Summary

Generating multiple-choice questions (MCQs) with high educational validity and psychometric quality—especially those grounded in authentic student misconceptions or conceptual understandings—remains challenging for large language models (LLMs). Method: We propose AnaQuest, a novel LLM-based framework that dynamically extracts concept-level correct and incorrect statements from students’ open-ended responses, integrating formative insights into summative item generation. The pipeline combines prompt engineering with open-text analysis and is rigorously validated using Item Response Theory (IRT) metrics and expert-rated content validity. Contribution/Results: AnaQuest significantly outperforms ChatGPT baselines across item difficulty, discrimination, and distractor quality. Its generated MCQs achieve psychometric properties and content validity comparable to human-authored items—the first end-to-end automated method to transform student cognitive traces into high-fidelity, psychometrically sound assessments.

Technology Category

• Cognitive Modeling & Cognitive Systems: Analogy
• Natural Language Processing: Question Answering
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Web Mining and Content Analysis: Community question answering
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The primary goal of this study is to develop and evaluate an innovative prompting technique, AnaQuest, for generating multiple-choice questions (MCQs) using a pre-trained large language model. In AnaQuest, the choice items are sentence-level assertions about complex concepts. The technique integrates formative and summative assessments. In the formative phase, students answer open-ended questions for target concepts in free text. For summative assessment, AnaQuest analyzes these responses to generate both correct and incorrect assertions. To evaluate the validity of the generated MCQs, Item Response Theory (IRT) was applied to compare item characteristics between MCQs generated by AnaQuest, a baseline ChatGPT prompt, and human-crafted items. An empirical study found that expert instructors rated MCQs generated by both AI models to be as valid as those created by human instructors. However, IRT-based analysis revealed that AnaQuest-generated questions - particularly those with incorrect assertions (foils) - more closely resembled human-crafted items in terms of difficulty and discrimination than those produced by ChatGPT.

Problem

• Develop AnaQuest for generating valid multiple-choice questions using LLMs
• Integrate formative and summative assessments via student understanding hints
• Compare AI-generated and human-crafted MCQs using Item Response Theory

Innovation

• AnaQuest generates MCQs from student responses
• Integrates formative and summative assessments
• Uses IRT to validate question quality