CyberSleuth: Autonomous Blue-Team LLM Agent for Web Attack Forensics

2025-08-28

AI Summary

Current LLM-based agent research in cybersecurity predominantly focuses on red-teaming tasks, while blue-team applications—particularly automated attack forensics—remain severely underexplored. Method: We propose CyberSleuth, the first systematic LLM agent framework for automated web-application attack forensics. It autonomously identifies attack targets, associated CVE vulnerabilities, and exploitation outcomes from network traffic and system logs, integrating LLM reasoning with tool-augmented execution. We comparatively evaluate four agent architectures and six LLM backends, and release the first reproducible adversarial benchmark platform for this task. Contribution/Results: CyberSleuth achieves state-of-the-art performance on 20 progressively complex forensic events and attains 80% CVE identification accuracy on 10 novel 2025 attacks. Expert evaluation confirms its reports exhibit completeness, operational utility, and logical rigor. This work bridges a critical gap in deploying defensive LLM agents for real-world digital forensics.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Adversarial Agents
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Large Language Model (LLM) agents are powerful tools for automating complex tasks. In cybersecurity, researchers have primarily explored their use in red-team operations such as vulnerability discovery and penetration tests. Defensive uses for incident response and forensics have received comparatively less attention and remain at an early stage. This work presents a systematic study of LLM-agent design for the forensic investigation of realistic web application attacks. We propose CyberSleuth, an autonomous agent that processes packet-level traces and application logs to identify the targeted service, the exploited vulnerability (CVE), and attack success. We evaluate the consequences of core design decisions - spanning tool integration and agent architecture - and provide interpretable guidance for practitioners. We benchmark four agent architectures and six LLM backends on 20 incident scenarios of increasing complexity, identifying CyberSleuth as the best-performing design. In a separate set of 10 incidents from 2025, CyberSleuth correctly identifies the exact CVE in 80% of cases. At last, we conduct a human study with 22 experts, which rated the reports of CyberSleuth as complete, useful, and coherent. They also expressed a slight preference for DeepSeek R1, a good news for open source LLM. To foster progress in defensive LLM research, we release both our benchmark and the CyberSleuth platform as a foundation for fair, reproducible evaluation of forensic agents.

Problem

• Autonomous agent for web attack forensic investigation
• Identifies targeted service, exploited vulnerability, and attack success
• Evaluates design decisions and benchmarks agent architectures

Innovation

• Autonomous agent processes packet traces and logs
• Identifies targeted service, exploited vulnerability, and attack success
• Benchmarks four agent architectures and six LLM backends