Decoupling Reconnaissance and Exploitation: Measuring the Capability Boundaries of LLM-Based Web Penetration Testing

2026-06-23

AI Summary

This work addresses a critical limitation in existing end-to-end black-box evaluations of large language models (LLMs) for automated exploitation, where errors in the reconnaissance phase obscure the true exploit capabilities of LLMs. To resolve this, the authors propose a two-stage decoupled evaluation framework that isolates reconnaissance and exploitation performance by injecting real-world vulnerability contexts and applying knowledge-driven ablation. Evaluated across 70 high-fidelity web vulnerability environments, the framework enables the first independent quantification of these two capabilities. Comparative analysis across 50 representative vulnerabilities reveals that, given accurate contextual information, LLMs achieve up to 90% exploit success rates, whereas autonomous reconnaissance yields only ~50% recall. Furthermore, multi-agent, monolithic, and graph-driven architectures exhibit distinct strengths and limitations across vulnerability types involving long-sequence interactions, short-chain injections, and cross-session access control, thereby delineating their respective capability boundaries.

Abstract

Large Language Models (LLMs) have shown promise for automated penetration testing, yet existing end-to-end black-box evaluations are highly susceptible to error cascading: failures in early reconnaissance can mask an agent's actual ability to exploit vulnerabilities. To more accurately characterize these capabilities, we propose a two-stage decoupled evaluation framework that separates exploit execution from reconnaissance. Using ground-truth injection and knowledge-driven ablation across 70 high-fidelity web vulnerability testbeds, our framework isolates exploitation performance from reconnaissance noise. We empirically evaluate five open-source penetration-testing agents, covering multiagent, monolithic, and graph-driven architectures, on a strictly aligned subset of 50 representative vulnerabilities. The results reveal a substantial capability gap. With accurate vulnerability context, agents achieve a functional success rate of up to 90.0%, whereas autonomous reconnaissance, measured by targeted vulnerability recall, plateaus at approximately 50.0%, primarily due to failures in parsing unstructured telemetry. Cross-architectural analysis further reveals distinct capability niches: multi-agent isolation is more effective for long-sequence interactions such as de-serialization, while monolithic and graph-driven designs perform better on short-chain injections and cross-session access-control vulnerabilities, respectively. This decoupled evaluation work provides a fine-grained benchmarking protocol and an empirical basis for designing next-generation automated offensive security agents.

Problem

• LLM-based penetration testing
• reconnaissance-exploitation decoupling
• capability evaluation
• vulnerability exploitation
• error cascading

Innovation

• decoupled evaluation
• LLM-based penetration testing
• reconnaissance-exploitation gap
• vulnerability exploitation
• automated offensive security