Starting Seatwork Earlier as a Valid Measure of Student Engagement

2025-05-19

AI Summary

Traditional assessments of self-regulated learning (SRL) and engagement often rely on intrusive, resource-intensive methods. This study investigates temporal decision-making behaviors—such as early start, delayed launch, and premature termination—extracted from learning platform session logs as unobtrusive, naturally embedded proxies for SRL and engagement. Method: Leveraging educational log analysis, generalizability theory (G-coefficient estimation), multiple regression, and incremental predictive modeling, the study conducts cross-platform validation across Cognitive Tutor and i-Ready. Contribution/Results: It provides the first systematic evidence that these behavioral indicators exhibit high month-to-month reliability (G > 0.75) and robust predictive validity: in Cognitive Tutor, they significantly improve prediction of end-of-term math performance beyond prior knowledge and cheating indicators; in i-Ready, they effectively predict seventh-grade state assessment outcomes. Critically, the approach requires no additional assessment administration, demonstrating strong cross-platform transferability and practical scalability for real-world educational settings.

Technology Category

• Machine Learning: Online Learning & Bandits
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Planning, Routing, and Scheduling: Model-Based Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Prior work has developed a range of automated measures ("detectors") of student self-regulation and engagement from student log data. These measures have been successfully used to make discoveries about student learning. Here, we extend this line of research to an underexplored aspect of self-regulation: students' decisions about when to start and stop working on learning software during classwork. In the first of two analyses, we build on prior work on session-level measures (e.g., delayed start, early stop) to evaluate their reliability and predictive validity. We compute these measures from year-long log data from Cognitive Tutor for students in grades 8-12 (N = 222). Our findings show that these measures exhibit moderate to high month-to-month reliability (G>.75), comparable to or exceeding gaming-the-system behavior. Additionally, they enhance the prediction of final math scores beyond prior knowledge and gaming-the-system behaviors. The improvement in learning outcome predictions beyond time-on-task suggests they capture a broader motivational state tied to overall learning. The second analysis demonstrates the cross-system generalizability of these measures in i-Ready, where they predict state test scores for grade 7 students (N = 818). By leveraging log data, we introduce system-general naturally embedded measures that complement motivational surveys without extra instrumentation or disruption of instruction time. Our findings demonstrate the potential of session-level logs to mine valid and generalizable measures with broad applications in the predictive modeling of learning outcomes and analysis of learner self-regulation.

Problem

• Measures student engagement via start/stop timing in classwork software
• Evaluates reliability and predictive validity of session-level engagement metrics
• Demonstrates cross-system generalizability of engagement measures for learning outcomes

Innovation

• Leveraging log data for student engagement measures
• Developing reliable session-level behavioral detectors
• Cross-system validation of predictive measures