From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning

2026-05-12

AI Summary

This study addresses insufficient learner engagement in intelligent tutoring systems by formulating weekly learning effort and skill mastery progression as supervised learning tasks. Leveraging interaction logs from 425 middle school students over an academic year, the authors construct and evaluate 15 regression, decision tree, and neural network models, establishing the first reproducible week-level prediction benchmark. Through feature importance analysis and ablation studies, they uncover distinct feature-driven mechanisms underlying different prediction targets. The proposed models reduce mean absolute error by 22–33% compared to heuristic baselines, effectively capturing students’ learning trajectories. User interviews further validate the models’ utility in supporting teachers’ goal-setting and instructional decision-making.

Abstract

Sustained effort is essential for realizing the benefits of intelligent tutoring systems (ITS), yet many learners disengage or underuse available practice time. We introduce engagement forecasting as a supervised prediction task based on ITS logs, targeting two outcomes central to effort and learning progress: minutes practiced per week and new skills mastered per week. Using interaction log data from 425 middle-school students over a school year, we benchmark fifteen predictors including regressions, decision trees, and neural networks. We show that these feature-based models reduce mean absolute error (MAE) by 22-33% relative to heuristic baselines, including fixed-percentile rules adapted from prior work in other behavioral domains. We find that percentile heuristics systematically overpredict, whereas feature-based models better track student practice trajectories across weeks. To support explainability, we analyze feature importance and ablations, revealing target-specific patterns: effort forecasting is driven mainly by recent activity features, while progress forecasting depends more on learner-state and content difficulty signals. Finally, in a semi-structured user interview case study with eight college tutors, we examine how tutors reasoned about system-generated predictive features when setting goals with students. We find that tutors reasoned differently about effort versus progress goals in ways that mirror our pattern analysis. Together, these results establish a reproducible benchmark for forecasting weekly effort and learning progress in ITS. By making patterns of sustained effort and progress visible at a weekly timescale, engagement forecasting offers a foundation for supporting tutor-learner goal setting and timely instructional decisions.

Problem

• engagement forecasting
• intelligent tutoring systems
• learning effort
• learning progress
• student disengagement

Innovation

• engagement forecasting
• intelligent tutoring systems
• learning analytics
• feature-based prediction
• explainable AI