A Human Study of Cognitive Biases in CTF Challenges

2025-05-17

AI Summary

This study investigates how cognitive biases—specifically search satisfaction and loss aversion—affect CTF participants’ ability to discover web vulnerabilities. Using a controlled human-subject experiment, we integrate quantitative behavioral tracking, qualitative task-based interviews, and a custom-built CTF platform featuring web applications with embedded vulnerabilities. Our findings provide the first empirical evidence that approximately 70% of participants exhibit significant search satisfaction bias, reducing flag discovery rates by an average of 25%; loss aversion further constrains attack strategy diversity. Crucially, we model cognitive biases as tunable, experimentally manipulable variables—enabling systematic investigation of their impact on security behavior. This work delivers key empirical evidence and a novel methodology to inform human-centered cybersecurity education, assessment design, and defensive strategy enhancement.

Technology Category

• Search and Optimization: Other Foundations of Search & Optimization
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Constraint Satisfaction and Optimization: Search

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

Cybersecurity training has become a crucial part of computer science education and industrial onboarding. Capture the Flag (CTF) competitions have emerged as a valuable, gamified approach for developing and refining the skills of cybersecurity and software engineering professionals. However, while CTFs provide a controlled environment for tackling real world challenges, the participants' decision making and problem solving processes remain under explored. Recognizing that psychology may play a role in a cyber attacker's behavior, we investigate how cognitive biases could be used to improve CTF education and security. In this paper, we present an approach to control cognitive biases, specifically Satisfaction of Search and Loss Aversion, to influence and potentially hinder attackers' effectiveness against web application vulnerabilities in a CTF style challenge. We employ a rigorous quantitative and qualitative analysis through a controlled human study of CTF tasks. CTF exercises are widely used in cybersecurity education and research to simulate real world attack scenarios and help participants develop critical skills by solving security challenges in controlled environments. In our study, participants interact with a web application containing deliberately embedded vulnerabilities while being subjected to tasks designed to trigger cognitive biases. Our study reveals that many participants exhibit the Satisfaction of Search bias and that this bias has a significant effect on their success. On average, participants found 25% fewer flags compared to those who did not exhibit this bias. Our findings provide valuable insights into how cognitive biases can be strategically employed to enhance cybersecurity outcomes, education, and measurements through the lens of CTF challenges.

Problem

• Investigates cognitive biases' impact on CTF performance
• Explores Satisfaction of Search bias in cybersecurity training
• Measures how biases reduce vulnerability detection effectiveness

Innovation

• Control cognitive biases in CTF challenges
• Quantitative and qualitative analysis of biases
• Strategic use of biases to enhance cybersecurity