CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution

2025-05-21

AI Summary

To address the critical bottlenecks of knowledge obsolescence and planning disconnection in large language models (LLMs) for cybersecurity tasks, this paper proposes CyberAgent, an agent framework tailored for adversarial cyber scenarios. Methodologically, it integrates three core innovations: (1) task-context decomposition to enable fine-grained, multi-stage planning for vulnerability detection and exploitation; (2) iterative self-reflective knowledge retrieval, dynamically fusing CTF-based operational knowledge with the MITRE ATT&CK framework; and (3) knowledge-augmented prompt injection, seamlessly incorporating retrieved knowledge into the reasoning process. The framework unifies retrieval-augmented generation (RAG), self-reflective reasoning, and ATT&CK alignment to enhance both knowledge freshness and strategic adaptability. Evaluated on NYU CTF Bench, CyberAgent achieves 22% accuracy—setting a new state-of-the-art (+3% over prior work)—and improves ATT&CK technique coverage by 25–30%. The implementation is publicly available.

Technology Category

• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Model (LLM) agents can automate cybersecurity tasks and can adapt to the evolving cybersecurity landscape without re-engineering. While LLM agents have demonstrated cybersecurity capabilities on Capture-The-Flag (CTF) competitions, they have two key limitations: accessing latest cybersecurity expertise beyond training data, and integrating new knowledge into complex task planning. Knowledge-based approaches that incorporate technical understanding into the task-solving automation can tackle these limitations. We present CRAKEN, a knowledge-based LLM agent framework that improves cybersecurity capability through three core mechanisms: contextual decomposition of task-critical information, iterative self-reflected knowledge retrieval, and knowledge-hint injection that transforms insights into adaptive attack strategies. Comprehensive evaluations with different configurations show CRAKEN's effectiveness in multi-stage vulnerability detection and exploitation compared to previous approaches. Our extensible architecture establishes new methodologies for embedding new security knowledge into LLM-driven cybersecurity agentic systems. With a knowledge database of CTF writeups, CRAKEN obtained an accuracy of 22% on NYU CTF Bench, outperforming prior works by 3% and achieving state-of-the-art results. On evaluation of MITRE ATT&CK techniques, CRAKEN solves 25-30% more techniques than prior work, demonstrating improved cybersecurity capabilities via knowledge-based execution. We make our framework open source to public https://github.com/NYU-LLM-CTF/nyuctf_agents_craken.

Problem

• Accessing latest cybersecurity expertise beyond training data
• Integrating new knowledge into complex task planning
• Improving cybersecurity capabilities via knowledge-based execution

Innovation

• Contextual decomposition of task-critical information
• Iterative self-reflected knowledge retrieval
• Knowledge-hint injection for adaptive strategies