Do Hackers Dream of Electric Teachers?: A Large-Scale, In-Situ Evaluation of Cybersecurity Student Behaviors and Performance with AI Tutors

2026-02-19

AI Summary

This study addresses the lack of empirical evidence on the impact of AI teaching assistants (AI TAs) in authentic, large-scale, hands-on cybersecurity courses centered on capture-the-flag (CTF) challenges. Through a semester-long field observation involving 309 students, the research analyzes 142,526 AI TA interaction logs across 396 challenges alongside post-course survey responses. It systematically identifies and categorizes three distinct conversational styles—terse, reactive, and proactive—and demonstrates that the proactive style significantly improves challenge completion rates, with stronger effects on more difficult tasks. While students generally perceive the AI TA as usable, its support for higher-order conceptual content remains limited. These findings offer empirical insights and practical guidance for the design and pedagogical integration of AI-powered educational tools in experiential learning environments.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

To meet the ever-increasing demands of the cybersecurity workforce, AI tutors have been proposed for personalized, scalable education. But, while AI tutors have shown promise in introductory programming courses, no work has evaluated their use in hands-on exploration and exploitation of systems (e.g., ``capture-the-flag'') commonly used to teach cybersecurity. Thus, despite growing interest and need, no work has evaluated how students use AI tutors or whether they benefit from their presence in real, large-scale cybersecurity courses. To answer this, we conducted a semester-long observational study on the use of an embedded AI tutor with 309 students in an upper-division introductory cybersecurity course. By analyzing 142,526 student queries sent to the AI tutor across 396 cybersecurity challenges spanning 9 core cybersecurity topics and an accompanying set of post-semester surveys, we find (1) what queries and conversational strategies students use with AI tutors, (2) how these strategies correlate with challenge completion, and (3) students' perceptions of AI tutors in cybersecurity education. In particular, we identify three broad AI tutor conversational styles among users: Short (bounded, few-turn exchanges), Reactive (repeatedly submitting code and errors), and Proactive (driving problem-solving through targeted inquiry). We also find that the use of these styles significantly predicts challenge completion, and that this effect increases as materials become more advanced. Furthermore, students valued the tutor's availability but reported that it became less useful for harder material. Based on this, we provide suggestions for security educators and developers on practical AI tutor use.

Problem

• AI tutors
• cybersecurity education
• student behavior
• large-scale evaluation
• hands-on learning

Innovation

• AI tutors
• cybersecurity education
• capture-the-flag
• student behavior analysis
• conversational strategies