Analysis of Log Data from an International Online Educational Assessment System: A Multi-state Survival Modeling Approach to Reaction Time between and across Action Sequence

2024-03-22

AI Summary

Modeling the temporal dynamics of problem-solving behaviors in online educational assessments remains challenging due to the complexity of sequential, time-stamped log data. Method: We propose a log analysis framework based on multi-state survival modeling—the first application of such models to educational assessment logs—explicitly capturing time-varying hazards and state-transition mechanisms between actions. We extend the Cox proportional hazards model to accommodate multi-stage operational sequences. Contribution/Results: Evaluated on PIAAC problem-solving item log data, our model reveals that the timing of key cognitive actions (e.g., information retrieval, solution verification) significantly influences subsequent transition rates and response accuracy. It achieves superior AIC fit compared to conventional survival and classification models, uncovering causal links between fine-grained behavioral timing patterns and final outcomes. This work establishes an interpretable, dynamic modeling paradigm for process-oriented assessment in educational measurement.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Planning, Routing, and Scheduling: Model-Based Reasoning
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Models for Web evolution
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

With increasingly available computer-based or online assessments, researchers have shown keen interest in analyzing log data to improve our understanding of test takers' problem-solving processes. In this paper, we propose a multi-state survival model (MSM) to action sequence data from log files, focusing on modeling test takers' reaction times between actions, in order to investigate which factors and how they influence test takers' transition speed between actions. In particular, we focus on the effects of the occurrence and timing of key actions that differentiate correct answers from incorrect answers. We demonstrate our proposed approach with problem-solving test items from the the Programme for International Assessment of Adult Competence (PIAAC) problem-solving test items.

Problem

• Model reaction times between actions in log data
• Identify factors influencing transition speed between actions
• Compare problem-solving patterns of correct vs incorrect answers

Innovation

• Multi-state survival model for log data
• Analyzes reaction times between actions
• Compares transition probabilities between groups