Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs

2025-10-20

AI Summary

This study empirically examines the relative efficacy of AI in cybersecurity offense versus defense, challenging the prevailing assumption that AI inherently favors attackers. Method: We design and implement CAI, a parallel-agent framework grounded in 23 real-world CTF scenarios, to systematically evaluate AI-driven attack success (initial access) and defense success (patch deployment), incorporating realistic operational constraints—including system availability and zero-intrusion guarantees. Contribution/Results: Under unconstrained conditions, AI-based defense significantly outperforms offense (54.3% vs. 28.3%, *p* < 0.01). However, when requiring both continuous system availability and provable intrusion prevention, the performance gap vanishes (23.9% vs. 15.2%, *p* > 0.05). This is the first controlled experimental demonstration that AI offensive–defensive efficacy differentials are critically contingent on success criteria. We propose a dual-dimension evaluation paradigm—operational feasibility *and* guaranteed security—to rigorously assess AI defense capability, establishing a novel benchmark for AI security evaluation.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Computer Vision: Adversarial Attacks & Robustness
• Humans and AI: AI for Accessibility

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We empirically evaluate whether AI systems are more effective at attacking or defending in cybersecurity. Using CAI (Cybersecurity AI)'s parallel execution framework, we deployed autonomous agents in 23 Attack/Defense CTF battlegrounds. Statistical analysis reveals defensive agents achieve 54.3% unconstrained patching success versus 28.3% offensive initial access (p=0.0193), but this advantage disappears under operational constraints: when defense requires maintaining availability (23.9%) and preventing all intrusions (15.2%), no significant difference exists (p>0.05). Exploratory taxonomy analysis suggests potential patterns in vulnerability exploitation, though limited sample sizes preclude definitive conclusions. This study provides the first controlled empirical evidence challenging claims of AI attacker advantage, demonstrating that defensive effectiveness critically depends on success criteria, a nuance absent from conceptual analyses but essential for deployment. These findings underscore the urgency for defenders to adopt open-source Cybersecurity AI frameworks to maintain security equilibrium against accelerating offensive automation.

Problem

• Evaluating AI effectiveness in cybersecurity attack versus defense scenarios
• Assessing defensive patching success versus offensive intrusion capabilities
• Analyzing how operational constraints impact AI defense performance metrics

Innovation

• Autonomous agents deployed in CTF battlegrounds
• Statistical analysis compares attack versus defense success
• Defensive effectiveness depends on operational constraints criteria