Discrepancies in Mental Workload Estimation: Self-Reported versus EEG-Based Measures in Data Visualization Evaluation

2025-07-12

AI Summary

Prior research lacks systematic comparison between subjective self-reports and neurophysiological measures of cognitive load (CL) in data visualization evaluation. Method: We conducted an experiment integrating visualization literacy assessment and spatial visualization tasks, simultaneously recording 32-channel EEG signals. A graph attention network (GAT) was developed to estimate mental workload (MW) from EEG, and results were compared against established subjective scales (e.g., NASA-TLX). Contribution/Results: Significant discrepancies emerged between EEG-derived MW estimates and subjective ratings across task difficulty levels. EEG proved sensitive to unconscious cognitive effort—unreported in self-assessments—and revealed dynamic CL fluctuations invisible to introspection. This study provides the first empirical validation in visualization research that neurophysiological metrics meaningfully complement subjective evaluation. It establishes a novel, objective, and fine-grained paradigm for CL assessment, offering both methodological innovation and theoretical grounding for advancing visualization usability evaluation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Data Mining & Knowledge Management: Data Visualization & Summarization
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Web Mining and Content Analysis: Web data visualization
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Accurate assessment of mental workload (MW) is crucial for understanding cognitive processes during visualization tasks. While EEG-based measures are emerging as promising alternatives to conventional assessment techniques, such as selfreport measures, studies examining consistency across these different methodologies are limited. In a preliminary study, we observed indications of potential discrepancies between EEGbased and self-reported MW measures. Motivated by these preliminary observations, our study further explores the discrepancies between EEG-based and self-reported MW assessment methods through an experiment involving visualization tasks. In the experiment, we employ two benchmark tasks: the Visualization Literacy Assessment Test (VLAT) and a Spatial Visualization (SV) task. EEG signals are recorded from participants using a 32-channel system at a sampling rate of 128 Hz during the visualization tasks. For each participant, MW is estimated using an EEG-based model built on a Graph Attention Network (GAT) architecture, and these estimates are compared with conventional MW measures to examine potential discrepancies. Our findings reveal notable discrepancies between task difficulty and EEG-based MW estimates, as well as between EEG-based and self-reported MW measures across varying task difficulty levels. Additionally, the observed patterns suggest the presence of unconscious cognitive effort that may not be captured by selfreport alone.

Problem

• Discrepancies between EEG-based and self-reported mental workload measures
• Inconsistencies in mental workload assessment during visualization tasks
• Unconscious cognitive effort not captured by self-report measures

Innovation

• EEG-based MW estimation using Graph Attention Network
• Comparison between EEG and self-reported MW measures
• Detection of unconscious cognitive effort via EEG