Automating Learner Assessment: Benchmarking Machine Learning and Deep Learning Models for EEG-Based Familiarity Prediction

2026-08-17

AI Summary

This study addresses the challenge of objectivity in educational assessment and critical data leakage issues in EEG-based familiarity prediction. We benchmarked 15 machine learning and deep learning models, introducing a rigorous Group K-Fold independent validation paradigm to correct performance inflation caused by temporal leakage. Integrating SHAP interpretability analysis, we established a corrected CNN F1-score of 0.6038 as a reliable evaluation baseline and identified frontotemporal Gamma/Beta oscillations as key cognitive biomarkers. By effectively mitigating model overestimation, this work provides a robust methodological framework and physiological evidence for neurocognitive monitoring in educational technology, ensuring more valid assessments of learning states through electrophysiological signals.

Abstract

Objective assessment of learning remains a fundamental challenge in education. Electroencephalography (EEG) provides a direct, non-invasive window into the neural correlates of knowledge acquisition, including cognitive familiarity. This study benchmarks fifteen machine learning (ML) and deep learning (DL) models for EEG-based familiarity prediction across two cognitive domains: faces (factual knowledge) and mathematical equations (conceptual knowledge). Using continuous EEG data from 23 participants, we extract spectral features (Power Spectral Density) across six frequency bands. We show that while standard stratified cross-validation yields artificially high classification performance (up to 0.9853 F1-score using CNN) due to temporal leakage across neighboring epochs, a rigorous trial-independent validation (Group K-Fold) drops the peak performance to 0.6038 F1-score (using CNN), which is still statistically significant above the 25% chance level. This highlights the critical necessity of trial-independent evaluation to avoid overestimating model generalizability. Furthermore, feature importance and SHAP analysis reveal that temporal and frontal Gamma and Beta oscillations are the most critical biomarkers for familiarity. This work establishes a realistic benchmark for EEG-based cognitive monitoring in educational technologies.

Problem

• Learner Assessment
• EEG
• Familiarity Prediction
• Benchmarking
• Evaluation Methodology

Innovation

• Trial-independent validation
• EEG familiarity prediction
• Temporal leakage
• SHAP analysis
• Cognitive monitoring benchmark