Detecting Unsuccessful Students in Cybersecurity Exercises in Two Different Learning Environments

2024-08-16

Frontiers in Education Conference

AI Summary

This study addresses early identification of learning risks in cybersecurity training. Leveraging operational logs from 313 students across two major cyber-range platforms—KYPO CRP and EDURange—we develop a cross-platform academic risk prediction model. Methodologically, we conduct the first comparative analysis of dual-platform feature engineering strategies to empirically validate the predictive validity and transferability of hands-on behavioral data across distinct learning environments. We extract multidimensional, fine-grained behavioral features and systematically evaluate eight classification algorithms, optimizing for both accuracy and sensitivity. Results show that the decision tree classifier achieves superior performance on both platforms (mean balanced accuracy >85%, sensitivity >82%), robustly identifying high-risk learners—particularly those with knowledge gaps or low frustration tolerance. All datasets and source code are publicly released to support evidence-based, targeted pedagogical interventions.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

This full paper in the research track evaluates the usage of data logged from cybersecurity exercises in order to predict students who are potentially at risk of performing poorly. Hands-on exercises are essential for learning since they enable students to practice their skills. In cybersecurity, hands-on exercises are often complex and require knowledge of many topics. Therefore, students may miss solutions due to gaps in their knowledge and become frustrated, which impedes their learning. Targeted aid by the instructor helps, but since the instructor's time is limited, efficient ways to detect struggling students are needed. This paper develops automated tools to predict when a student is having difficulty. We formed a dataset with the actions of 313 students from two countries and two learning environments: KYPO CRP and EDURange. These data are used in machine learning algorithms to predict the success of students in exercises deployed in these environments. After extracting features from the data, we trained and cross-validated eight classifiers for predicting the exercise outcome and evaluated their predictive power. The contribution of this paper is comparing two approaches to feature engineering, modeling, and classification performance on data from two learning environments. Using the features from either learning environment, we were able to detect and distinguish between successful and struggling students. A decision tree classifier achieved the highest balanced accuracy and sensitivity with data from both learning environments. The results show that activity data from cybersecurity exercises are suitable for predicting student success. In a potential application, such models can aid instructors in detecting struggling students and providing targeted help. We publish data and code for building these models so that others can adopt or adapt them.

Problem

• Predicting at-risk students in cybersecurity exercises using logged data.
• Developing automated tools to detect struggling students efficiently.
• Comparing feature engineering and classification across two learning environments.

Innovation

• Automated tools predict student difficulty
• Machine learning analyzes cybersecurity exercise data
• Decision tree classifier achieves highest accuracy