SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications

2026-05-10

AI Summary

This study addresses the absence of a systematic benchmark for evaluating the quality of Solidity smart contracts generated by large language models (LLMs). We present the first comprehensive benchmark comprising 9,000 LLM-generated contracts paired with expert-written reference implementations. To enable rigorous assessment, we propose a five-dimensional evaluation framework encompassing functional completeness, state machine correctness, and other critical criteria, alongside a reproducible generation and evaluation pipeline that integrates Slither static analysis, manual review, and automated scoring. Empirical analysis reveals prevalent failure modes—notably logic omissions (35.3%) and state transition errors (23.4%)—and shows that generated contracts score on average 8.29 points higher than reference implementations in overall quality. All data and code are publicly released to support future research.

Abstract

We introduce SmartEval, a benchmark for systematically evaluating the quality of Solidity smart contracts generated by large language models (LLMs) from natural language specifications. SmartEval provides a corpus of 9,000 generated contracts paired with expert-written ground-truth implementations drawn from the FSMSCG dataset, a five-dimensional evaluation rubric covering functional completeness, variable fidelity, state-machine correctness, business-logic fidelity, and code quality, and a reproducible generation-and-evaluation pipeline. To validate the benchmark's reliability, we conduct three independent empirical studies: a five-condition ablation study (N=300 per condition) isolating the contribution of each pipeline component, a human expert evaluation by three Columbia University PhD researchers confirming automated scores align with expert judgment to within 0.34 points, and external security analysis via the Slither static analyzer confirming 79.4% agreement between the LLM auditor and a non-LLM rule-based tool. Systematic analysis of 9,000 generated contracts reveals characteristic failure modes (logic omissions at 35.3%, state transition errors at 23.4%, and complexity-driven degradation) and quantifies a +8.29 composite-score advantage of generated contracts over ground-truth implementations, attributable to LLMs' literal specification-following behavior. SmartEval establishes a reproducible, validated foundation for empirical research on LLM smart contract synthesis quality, with all data, evaluation code, and generated contracts publicly released.

Problem

• smart contracts
• large language models
• natural language specifications
• evaluation benchmark
• code generation

Innovation

• SmartEval
• LLM-generated smart contracts
• evaluation benchmark
• natural language to Solidity
• empirical validation