APT-Agent: Automated Penetration Testing using Large Language Models

2026-05-24

AI Summary

This work addresses the growing inadequacy of manual penetration testing in the face of increasingly complex modern web infrastructures and the limitations of existing large language model (LLM)-driven automation, which often suffers from technical entity hallucination and insufficient long-term contextual memory. To overcome these challenges, the authors propose a fully automated LLM-based penetration testing framework that systematically orchestrates reconnaissance, exploitation, and data exfiltration phases. The framework incorporates a hybrid correction mechanism to suppress command hallucinations and introduces a command-level memory architecture to preserve contextual coherence across multi-step attacks. Evaluated on seven vulnerable services spanning web applications, databases, and network protocols within the Metasploitable 2 platform, the approach achieves an end-to-end attack success rate of 84.29%, substantially outperforming both Script Kiddie (48.57%) and PentestGPT (18.57%).

Abstract

Penetration testing is essential to securing modern web infrastructures, yet traditional manual methods struggle to keep pace with their scale and complexity. Large Language Models (LLMs) offer new opportunities for automating these tasks, but existing approaches face two persistent challenges: hallucination of technical entities and insufficient long-term contextual memory. To address these issues, we present APT-Agent, a fully automated LLM-driven penetration testing framework that systematically orchestrates reconnaissance, exploitation, and exfiltration. APT-Agent introduces a hybrid rectification module to recover hallucinated commands and a command-specific memory architecture to preserve operational context across multi-step attack sequences. We evaluate our APT-Agent on Metasploitable 2 against seven vulnerable services spanning web, database, and network protocols. APT-Agent achieves an 84.29% end-to-end exploitation success rate, compared to 48.57% (Script Kiddie) and 18.57% (PentestGPT) under matched conditions. By reducing cognitive burden and minimizing reliance on human intervention, APT-Agent represents a step toward scalable, reliable, and cognitively efficient automation for penetration testing.

Problem

• penetration testing
• Large Language Models
• hallucination
• contextual memory
• automation

Innovation

• Large Language Models
• Automated Penetration Testing
• Hallucination Rectification
• Contextual Memory Architecture
• Exploitation Automation