An end-to-end agentic pipeline for smart contract translation and quality evaluation

2026-02-14

AI Summary

This work addresses the lack of systematic evaluation of large language models (LLMs) in generating smart contracts from natural language, which hinders reliable assessment of their functional correctness, security, and logical consistency. To bridge this gap, we propose an end-to-end multi-agent evaluation framework built on CrewAI that enables collaborative optimization across natural language parsing, structured modeling, and Solidity code generation. The framework introduces a five-dimensional quality metric—encompassing functional completeness, variable fidelity, state machine correctness, among others—and incorporates traceable metadata. Through integrated compilation validation, static security analysis, and pairwise comparison against real-world contracts, our approach reproducibly identifies logical omissions and state transition errors, establishing the first systematic benchmark for automated smart contract generation.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

We present an end-to-end framework for systematic evaluation of LLM-generated smart contracts from natural-language specifications. The system parses contractual text into structured schemas, generates Solidity code, and performs automated quality assessment through compilation and security checks. Using CrewAI-style agent teams with iterative refinement, the pipeline produces structured artifacts with full provenance metadata. Quality is measured across five dimensions, including functional completeness, variable fidelity, state-machine correctness, business-logic fidelity, and code quality aggregated into composite scores. The framework supports paired evaluation against ground-truth implementations, quantifying alignment and identifying systematic error modes such as logic omissions and state transition inconsistencies. This provides a reproducible benchmark for empirical research on smart contract synthesis quality and supports extensions to formal verification and compliance checking.

Problem

• smart contract
• quality evaluation
• LLM-generated code
• natural-language specification
• contract synthesis

Innovation

• agentic pipeline
• smart contract synthesis
• LLM evaluation
• structured schema extraction
• automated quality assessment