Multimodal Analytics of Cybersecurity Crisis Preparation Exercises: What Predicts Success?

2026-03-30

AI Summary

This study addresses the need to quantify the alignment between instructional objectives and actual activities in cybersecurity exercises and examines its predictive power for team performance. Leveraging multimodal data—comprising email communications and system logs—from 23 student teams across five exercises, we model task cognitive levels using Bloom’s Taxonomy and introduce a novel “instructional alignment” metric. We further develop a multimodal predictive model that integrates text embeddings with behavioral logs. Model evaluation employs generalized linear mixed models, L1-regularized logistic regression, and grouped cross-validation. Results demonstrate that the multimodal model significantly outperforms a baseline relying solely on Bloom-derived features (test AUC: 0.80 vs. 0.55), confirming the critical predictive value of instructional alignment and yielding interpretable diagnostic insights into team dynamics and learning outcomes.

Abstract

Instructional alignment, the match between intended cognition and enacted activity, is central to effective instruction but hard to operationalize at scale. We examine alignment in cybersecurity simulations using multimodal traces from 23 teams (76 students) across five exercise sessions. Study 1 codes objectives and team emails with Bloom's taxonomy and models the completion of key exercise tasks with generalized linear mixed models. Alignment, defined as the discrepancy between required and enacted Bloom levels, predicts success, whereas the Bloom category alone does not predict success once discrepancy is considered. Study 2 compares predictive feature families using grouped cross-validation and l1-regularized logistic regression. Text embeddings and log features outperform Bloom-only models (AUC~0.74 and 0.71 vs. 0.55), and their combination performs best (Test AUC~0.80), with Bloom frequencies adding little. Overall, the work offers a measure of alignment for simulations and shows that multimodal traces best forecast performance, while alignment provides interpretable diagnostic insight.

Problem

• instructional alignment
• cybersecurity simulations
• multimodal analytics
• Bloom's taxonomy
• performance prediction

Innovation

• multimodal analytics
• instructional alignment
• Bloom's taxonomy
• cybersecurity simulation
• predictive modeling