CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities?

2026-06-24

AI Summary

This study addresses the absence of a unified benchmark for evaluating AI agents’ capabilities in detecting, exploiting, and patching smart contract vulnerabilities within real on-chain environments. The authors present the first end-to-end evaluation framework grounded in 541 real-world DeFi attack incidents, encompassing vulnerability detection, exploit generation, and patch synthesis. By leveraging mainnet forking and isolated environments, the framework faithfully reproduces historical on-chain states. Key innovations include a structured evaluation suite covering multi-chain attack scenarios, an economic-impact-driven exploit scoring mechanism, a transaction-replay-based patch validation method, and a novel five-category vulnerability taxonomy. Experiments demonstrate that the best-performing agent (Codex+GPT-5.5) achieves potential gains of $57.4 million across 200 cases at a marginal cost of $2.39 per instance, with detection, exploitation, and patching accuracies of 37.5%, 43.7%, and 23.4%, respectively.

Abstract

We present CyberChainBench, a benchmark for evaluating LLM-based agents on smart contract security across three complementary tasks: vulnerability detection, exploit generation, and patch synthesis. Built from 541 real-world exploit incidents from DeFiHackLabs spanning 9 EVM chains, the benchmark provides end-to-end on-chain evaluation where agents interact with historical blockchain state through isolated evaluation environments orchestrated by Harbor, using tools to read code, trace transactions, and validate exploits on mainnet forks. Each case is anchored to a specific block and includes structured ground truth covering vulnerability type, localization, and attacker profit. Exploits are graded by economic impact on historical forks; patches are validated by replaying historical attacks and legitimate transactions as fail-to-pass test oracles on a proxy-upgradeable subset. We define a five-type vulnerability taxonomy and evaluate multiple agent--model configurations. Results reveal a clear difficulty gradient: the best configuration scores 37.5% on detection, 43.7% on exploitation, but only 23.4% on patching, with the top agent (Codex with GPT-5.5) realizing \$57.4M in total exploit profit across the 200-case exploit set at a cost of $2.39 per case.

Problem

• smart contract security
• LLM-based agents
• on-chain vulnerabilities
• vulnerability detection
• patch synthesis

Innovation

• smart contract security
• LLM agents
• on-chain evaluation
• exploit generation
• patch synthesis