Evaluation of LLM-based Explanations for a Learning Analytics Dashboard

2025-11-11

AI Summary

Learning analytics dashboards often lack interpretability, hindering students’ self-regulated learning and metacognitive development. Method: This study proposes a large language model (LLM)-based natural language explanation generation method to enhance dashboard interpretability. Through an empirical comparative experiment, LLM-generated skill-state interpretations and personalized learning recommendations were evaluated by domain experts alongside teacher-crafted explanations and a no-explanation baseline. Contribution/Results: This work presents the first systematic validation in educational technology demonstrating that LLM-generated explanations significantly outperform baselines in pedagogical appropriateness, comprehensibility, and practical utility—particularly in diagnostic accuracy, recommendation feasibility, and overall educational value—as rated by education experts. Findings confirm that expert-guided LLMs serve as reliable tools for improving dashboard explainability and pedagogical support. The study establishes a novel paradigm for explainable AI in learning analytics, advancing both theoretical understanding and practical implementation of interpretable educational technologies.

Technology Category

• Machine Learning: Transparent, Interpretable, Explainable ML
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Learning Analytics Dashboards can be a powerful tool to support self-regulated learning in Digital Learning Environments and promote development of meta-cognitive skills, such as reflection. However, their effectiveness can be affected by the interpretability of the data they provide. To assist in the interpretation, we employ a large language model to generate verbal explanations of the data in the dashboard and evaluate it against a standalone dashboard and explanations provided by human teachers in an expert study with university level educators (N=12). We find that the LLM-based explanations of the skill state presented in the dashboard, as well as general recommendations on how to proceed with learning within the course are significantly more favored compared to the other conditions. This indicates that using LLMs for interpretation purposes can enhance the learning experience for learners while maintaining the pedagogical standards approved by teachers.

Problem

• Evaluating LLM-generated explanations for learning analytics dashboard data
• Assessing interpretability of dashboard data to support self-regulated learning
• Comparing LLM explanations against human teacher and standalone dashboard approaches

Innovation

• LLM generates verbal explanations for dashboard data
• LLM provides skill state interpretations and learning recommendations
• LLM explanations enhance learning experience while maintaining standards