Transferable Self-Evolving Playbooks for Agentic Security Auditing

2026-06-15

AI Summary

This work addresses the limitations of traditional security auditing, which relies on manually crafted playbooks that are difficult to automate and lack cross-agent transferability. The authors propose EvoHunt, a novel framework that enables the first fully automated evolution and cross-model transfer of security audit playbooks. EvoHunt employs three types of agents—auditing, evaluation, and revision—that collaboratively optimize playbooks in a closed loop over open-source repositories without human intervention. Integrating large language models, an agent execution framework, and an evolvable playbook structure, the approach significantly enhances the vulnerability discovery and validation capabilities of weaker models. Experiments demonstrate that evolved playbooks improve end-to-end exploit success rates by 6× for Codex/GPT-5.4-xhigh and, when transferred to Qwen3.6-27B, increase target match rates from 2.4% to 6.5%, surpassing existing commercial solutions.

Abstract

An LLM agent for vulnerability discovery and validation is more than a model. It combines three components: an LLM for code analysis, an agent harness such as Codex or OpenCode for navigation, tool use, and execution, and an audit playbook, domain-specific procedural knowledge that guides the LLM and harness toward vulnerability discovery. Prior work relies on human-supplied playbooks, including prompt engineering, manual workflows, knowledge bases, and heuristics. This raises two research questions: Acquisition - is human curation necessary, and can playbook creation be automated? Transfer - can an evolved playbook transfer the audit procedure to weaker agents, improving their capability? We present EvoHunt, a playbook evolution environment over open-source repositories for security auditing. Three agents drive the evolution loop: an audit agent rolls out the current playbook and produces findings; an evaluator scores outcomes against ground truth; and a reviser commits updates to the playbook based on failure analysis. The playbook format is unconstrained: starting empty, EvoHunt adds or removes workflows, heuristics, vulnerability knowledge, or domain-specific content. The evolved playbook requires only minor adaptation to run under a different LLM or harness. We evaluate EvoHunt on open-source security advisories. For acquisition, playbook evolution raises end-to-end exploits for Codex/GPT5.4-xhigh 6x, from 1.1% to 6.2%, and the evolved OpenCode/GLM5.1 playbook surpasses OpenAI Codex Security on every metric, with 11.3% vs. 9.2% target-match rate, showing open-source evolution can outperform a dedicated commercial product. For transfer, the GLM-evolved playbook gives the strongest student lift: Qwen3.6-27B improves from 2.4% to 6.5%, Qwen3.6-35B-A3B from 1.1% to 4.6%, and A3B obtains 2.4x more matches than GPT transfer.

Problem

• playbook evolution
• agentic security auditing
• transferable audit procedures
• automated vulnerability discovery
• LLM agent

Innovation

• playbook evolution
• agentic security auditing
• transferable audit procedures
• automated vulnerability discovery
• LLM agent collaboration