Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning

2026-06-13

AI Summary

This study addresses the challenge of effectively evaluating and enhancing the capability of large language models (LLMs) to generate high-quality proof-of-concept (PoC) exploit code from CVE vulnerability descriptions. The authors propose a data-driven paradigm that constructs a high-quality exploit dataset through multi-stage data curation and introduces a scalable evaluation framework based on LLM-as-a-judge with fine-grained scoring criteria. They systematically assess the zero-shot performance of 17 LLMs and investigate the impact of instruction tuning and test-time rejection strategies. Experimental results demonstrate that an instruction-tuned 8B open-source model achieves over a 42.5% improvement in PoC generation quality, and when combined with a simple rejection mechanism, its performance rivals that of certain closed-source models, underscoring the critical role of data quality and thoughtful evaluation design in cybersecurity tasks.

Abstract

We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context. We adopt a data-centric approach, constructing a high-quality dataset via multi-stage preprocessing and introducing a scalable evaluation framework with LLM-as-judge and fine-grained rubrics. Under this unified setup, we benchmark 17 large language models across 8 evaluation criteria, providing systematic insights into their zero-shot capabilities. We further show that a compact 8B open-weight model, when fine-tuned on curated data, achieves over 42.5% improvement in exploit quality and rivals some proprietary models when combined with simple test-time rejection strategies. Our results highlight the importance of data quality, structured supervision, and evaluation design for reliable exploit generation, suggesting that these factors can be as critical as model scale in adapting LLMs to cybersecurity tasks.

Problem

• exploit generation
• CVE-conditioned
• data-centric
• large language models
• cybersecurity

Innovation

• data-centric
• exploit generation
• fine-tuning
• LLM-as-judge
• vulnerability