D-CIPHER: Dynamic Collaborative Intelligent Agents with Planning and Heterogeneous Execution for Enhanced Reasoning in Offensive Security

2025-02-15

AI Summary

解决单智能体在复杂网络安全任务中表现不足的问题，提出D-CIPHER多智能体框架，通过规划者和执行者角色分工及动态反馈提升CTF挑战解决效率。

Technology Category

• Multiagent Systems: Adversarial Agents
• Intelligent Robots: Multi-Robot Systems
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Security and Privacy: Applications of cryptography
• Responsible Web: Measurement, analysis, and circumvention of Web censorship
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large Language Models (LLMs) have been used in cybersecurity in many ways, including their recent use as intelligent agent systems for autonomous security analysis. Capture the Flag (CTF) challenges serve as benchmarks for assessing the automated task-planning abilities of LLM agents across various cybersecurity skill sets. Early attempts to apply LLMs for solving CTF challenges relied on single-agent systems, where feedback was restricted to a single reasoning-action loop. This approach proved inadequate for handling complex CTF tasks. Drawing inspiration from real-world CTF competitions, where teams of experts collaborate, we introduce the D-CIPHER multi-agent LLM framework for collaborative CTF challenge solving. D-CIPHER integrates agents with distinct roles, enabling dynamic feedback loops to enhance reasoning on CTF challenges. It introduces the Planner-Executor agent system, consisting of a Planner agent for overall problem-solving along with multiple heterogeneous Executor agents for individual tasks, facilitating efficient allocation of responsibilities among the LLMs. Additionally, D-CIPHER incorporates an Auto-prompter agent, which improves problem-solving by exploring the challenge environment and generating a highly relevant initial prompt. We evaluate D-CIPHER on CTF benchmarks using multiple LLM models and conduct comprehensive studies to highlight the impact of our enhancements. Our results demonstrate that the multi-agent D-CIPHER system achieves a significant improvement in challenges solved, setting a state-of-the-art performance on three benchmarks: 22.0% on NYU CTF Bench, 22.5% on Cybench, and 44.0% on HackTheBox. D-CIPHER is available at https://github.com/NYU-LLM-CTF/nyuctf_agents as the nyuctf_multiagent package.

Problem

• Enhances LLM reasoning in cybersecurity tasks
• Introduces multi-agent framework for CTF challenges
• Improves task-solving efficiency via role allocation

Innovation

• Multi-agent framework for cybersecurity tasks
• Planner-Executor system for task allocation
• Auto-prompter agent enhances initial prompts