penetration testing

Security practitioners face challenges in assessing exploit samples from public vulnerability databases due to inconsistent sample quality and inefficient evaluation: existing metrics such as CVSS and EPSS fail to reflect practical exploitability, while manual triage is time-consuming and unscalable. This paper introduces the first automated scoring system for exploit actionability—i.e., the feasibility of deploying an exploit in real-world environments—integrating static code analysis with multi-dimensional feature modeling. The system generates interpretable, structured quantitative scores across three dimensions: usability, functional correctness, and deployment complexity, explicitly accounting for both exploit logic and environmental dependencies—avoiding opaque probabilistic estimation. Evaluated on a dataset of over 5,000 vulnerabilities, the approach achieves 100% top-3 recommendation accuracy and strong agreement with expert assessments, significantly outperforming CVSS and EPSS.

To address the limited operational context adaptability of existing information system security assessments, this study proposes a实战-oriented, full-lifecycle penetration testing methodology. The approach systematically integrates five core phases: reconnaissance, vulnerability scanning, exploitation, privilege escalation, and post-exploitation. Crucially, it tightly couples penetration activities with frontline network operations, introducing a reusable, standardized operational framework and a risk-closure validation mechanism. Leveraging mainstream tools—including Nmap, Metasploit, and Burp Suite—the methodology incorporates active reconnaissance, automated exploitation, simulated lateral movement, and forensic log analysis. Validated end-to-end in real-world enterprise environments, the framework identified 12 categories of critical vulnerabilities, achieving a mean detection accuracy of 93.7%. Results demonstrate significant improvements in security incident response efficiency and defensive effectiveness.

This work addresses a critical limitation in existing end-to-end black-box evaluations of large language models (LLMs) for automated exploitation, where errors in the reconnaissance phase obscure the true exploit capabilities of LLMs. To resolve this, the authors propose a two-stage decoupled evaluation framework that isolates reconnaissance and exploitation performance by injecting real-world vulnerability contexts and applying knowledge-driven ablation. Evaluated across 70 high-fidelity web vulnerability environments, the framework enables the first independent quantification of these two capabilities. Comparative analysis across 50 representative vulnerabilities reveals that, given accurate contextual information, LLMs achieve up to 90% exploit success rates, whereas autonomous reconnaissance yields only ~50% recall. Furthermore, multi-agent, monolithic, and graph-driven architectures exhibit distinct strengths and limitations across vulnerability types involving long-sequence interactions, short-chain injections, and cross-session access control, thereby delineating their respective capability boundaries.

This work addresses the limitations of existing adversarial simulation tools, which rely on agent-based instrumentation of target systems, often leaving anomalous artifacts and failing to faithfully replicate human attacker behavior—particularly in critical phases of the cyber kill chain such as initial access and interactive operations. To overcome these shortcomings, the authors propose and implement an open-source attack scripting language coupled with an agentless execution engine that closely emulates real-world attacker tactics. This approach enables high-fidelity, interactive simulation of complete kill chain stages, including initial access, privilege escalation, and lateral movement. Experimental results demonstrate that system logs generated by this method exhibit significantly greater behavioral similarity to those produced by actual human-driven attacks, thereby enhancing the realism and effectiveness of security testing and intrusion detection research.

To address the challenge of integrating automation with expert judgment in penetration testing, this paper proposes a lightweight, real-time, and interpretable large language model (LLM)-assisted security assessment framework. Methodologically, it introduces a novel chain-of-thought compression mechanism and designs a retrieval-augmented generation (RAG) architecture tailored for security documentation, integrating semantic understanding of binaries and configuration files with multimodal parsing, and enabling end-to-end edge inference within browsers via WebAssembly. The contributions are threefold: (1) bridging the capability gap between automated tools and human experts, thereby lowering the barrier to advanced offensive and defensive operations; (2) significantly improving vulnerability identification accuracy and report generation efficiency; and (3) reducing repetitive document analysis time by 70%, decreasing hallucination rates by 52%, and—uniquely—supporting full-cycle foundational penetration testing directly in the browser.

Security teams struggle to patch all newly disclosed vulnerabilities in a timely manner and urgently need to prospectively assess exploitability under limited computational resources while avoiding assessment bias caused by data leakage. This work proposes a leakage-resistant prospective evaluation protocol that constructs a temporal evidence graph integrating vulnerability advisories, exploits, patch commits, and community discussions. Under computational budget constraints, the method selects critical evidence to generate auditable evidence certificates supporting contestable prioritization decisions. Experiments on 12,012 CVEs demonstrate that using only two pieces of evidence increases leakage-resistant recall@50 from 0.010 to 0.026, while also revealing that semantic similarity does not equate to valid exploitability evidence.

This work addresses the growing gap between the surge in vulnerability disclosures and organizations’ capacity to assess them, exacerbated by the longstanding fragmentation in research on exploit generation, prioritization, and detection rule engineering. To bridge this divide, the authors propose FORGE, a multi-agent system that introduces “exploit depth tiers” as a unifying framework for end-to-end vulnerability assessment. FORGE constructs targeted applications from CVE metadata, employs LLM-guided four-tier evaluation to iteratively refine exploits, and automatically generates Sigma and Snort detection rules from OpenTelemetry behavioral traces. Evaluated on 603 CVEs, FORGE achieves a 67.8% end-to-end success rate for L1+ exploits at a cost of $1.50 per CVE, produces L2+ detection rules that significantly improve normalized coverage (p=0.035), and yields Snort rules with zero false positives on benign traffic in 93.4% of cases.

This work addresses the limitations of traditional approaches in determining whether library versions are affected by vulnerabilities, which often suffer from mislabeling or missed detections and rely heavily on costly manual analysis. The authors propose a novel method that, for the first time, integrates execution trace differences from exploit attempts with the tool-calling capabilities of large language models (LLMs). Through a three-module architecture—comprising trace construction, difference exploration, and impact assessment—the approach guides the model to automatically analyze cross-version code changes and reason about vulnerability impact. It effectively handles challenges such as failed exploits and ambiguous commit messages, achieving an F1 score of 93.24% on a dataset of 224 CVEs and 25,943 library versions, significantly outperforming baseline methods like V-SZZ and LLM4SZZ.

This study investigates whether large language model (LLM)–assisted vulnerability repair genuinely enhances patch quality in real-world scenarios or merely produces “superficial fixes” that pass functional tests but fail security validation. To address this, we design a balanced, crossover human-subject experiment comparing developers’ performance—with and without LLM assistance—in terms of repair speed, functional correctness, and security robustness. We introduce, for the first time, a hidden “Ghost Tests” mechanism to evaluate patch completeness, innovatively integrating human-centered research with covert security assessments to systematically measure the actual security efficacy of LLM-assisted repair. Preliminary pilot results suggest that while LLMs can accelerate the repair process, they may also introduce subtle security risks; definitive conclusions await validation from the full-scale main experiment.

This work addresses the lack of systematic evaluation benchmarks for large language models (LLMs) in security audit log investigation tasks by introducing AuditBench, the first audit log benchmark specifically designed for attack investigation. AuditBench encompasses over 50 real-world scenarios across Linux and Windows systems and focuses on four core tasks: alert classification, persistence mechanism identification, among others. Through multidimensional experiments, the study systematically evaluates the impact of model scale, log representation, prompt design, and fine-tuning strategies on performance and error patterns, while also analyzing the quality of LLM-generated explanations. The findings reveal the capability boundaries and characteristic failure modes of various models across different investigative tasks, providing empirical foundations for deploying and optimizing LLMs in security operations.