Early-Warning Learner Satisfaction Forecasting in MOOCs via Temporal Event Transformers and LLM Text Embeddings

2026-04-14

AI Summary

This study addresses the limitation of existing approaches that rely on post-course ratings to predict learner satisfaction, thereby hindering timely intervention. To enable early prediction, the authors propose the TET-LLM framework, which integrates multimodal signals—including learners’ temporal behavioral sequences, large language model (LLM) embeddings of textual traces, and topic distributions—during the initial phase of a course. The framework innovatively combines a Temporal Event Transformer with LLM-derived embeddings and incorporates a heteroscedastic regression head to quantify prediction uncertainty. Evaluated under a 7-day prediction window, the model achieves an RMSE of 0.82 and an AUC of 0.77, significantly outperforming baseline methods, while its prediction intervals demonstrate strong calibration.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Learner satisfaction is a critical quality signal in massive open online courses (MOOCs), directly influencing retention, engagement, and platform reputation. Most existing methods infer satisfaction \emph{post hoc} from end-of-course reviews and star ratings, which are too late for effective intervention. In this paper, we study \textbf{early-warning satisfaction forecasting}: predicting a learner's eventual satisfaction score using only signals observed in the first $t$ days of a course (e.g., $t\!\in\!\{7, 14, 28\}$). We propose \textbf{TET-LLM}, a multi-modal fusion framework that combines (i) a \emph{temporal event Transformer} over fine-grained behavioral event sequences, (ii) \emph{LLM-based contextual embeddings} extracted from early textual traces such as forum posts and short feedback, and (iii) short-text \emph{topic/aspect distributions} to capture coarse satisfaction drivers. A heteroscedastic regression head outputs both a point estimate and a predictive uncertainty score, enabling conservative intervention policies. Comprehensive experiments on a large-scale multi-platform MOOC dataset demonstrate that TET-LLM consistently outperforms aggregate-feature and text-only baselines across all early-horizon settings, achieving an RMSE of 0.82 and AUC of 0.77 at the 7-day horizon. Ablation studies confirm the complementary contribution of each modality, and uncertainty calibration analysis shows near-nominal 90\% interval coverage.

Problem

• learner satisfaction
• early-warning forecasting
• MOOCs
• temporal event
• LLM embeddings

Innovation

• Temporal Event Transformer
• LLM Text Embeddings
• Early-Warning Forecasting
• Multi-modal Fusion
• Predictive Uncertainty