🤖 AI Summary
In supervised cybersecurity CTF training, evaluating learning outcomes and identifying flaws in training design remain challenging. To address these issues, this paper proposes an evaluation framework integrating process mining with multidimensional visual analytics. It models participants’ operational behavior sequences and implements an open-source, interactive dashboard supporting temporal pattern recognition, multivariate network visualization, and clustering analysis—rigorously adhering to established visualization design principles. Our key innovation lies in deeply embedding process mining into cybersecurity pedagogical assessment, enabling automated discovery of process deviations, bottlenecks, and organizational anomalies directly from system logs. A case study demonstrates that the framework effectively quantifies learner engagement, pinpoints training deficiencies—including task bottlenecks and imbalanced resource allocation—and substantially enhances the interpretability of evaluation results and their utility for instructional improvement.
📝 Abstract
Hands-on training sessions become a standard way to develop and increase knowledge in cybersecurity. As practical cybersecurity exercises are strongly process-oriented with knowledge-intensive processes, process mining techniques and models can help enhance learning analytics tools. The design of our open-source analytical dashboard is backed by guidelines for visualizing multivariate networks complemented with temporal views and clustering. The design aligns with the requirements for post-training analysis of a special subset of cybersecurity exercises -- supervised Capture the Flag games. Usability is demonstrated in a case study using trainees' engagement measurement to reveal potential flaws in training design or organization.