Repository-Level Solidity Code Generation with Large Language Models: From Prompting to Fine-Tuning

2026-06-18

AI Summary

Large language models struggle to simultaneously satisfy linguistic correctness, security requirements, and software engineering constraints when generating repository-scale Solidity smart contracts, and lack domain-specific evaluation benchmarks. This work introduces SolidityBench, the first comprehensive benchmark for repository-level Solidity code generation, along with SolidityScore, a novel semantic evaluation metric. The study systematically evaluates prominent models—including Qwen2.5-Coder, DeepSeek-Coder, and CodeLlama—under various prompting and adaptation strategies such as zero-shot prompting, chain-of-thought reasoning, in-context learning, retrieval-augmented generation, and supervised fine-tuning. Experimental results demonstrate that supervised fine-tuning with high-quality domain-specific data is most effective, successfully internalizing Solidity constraints into model parameters. Among non-parametric approaches, retrieval-augmented generation yields the best performance, while in-context learning degrades beyond two examples due to context saturation.

Abstract

Large Language Models (LLMs) have shown strong capabilities in general-purpose code generation, but their effectiveness in specialized software domains remains underexplored. Solidity smart contracts represent a high-stakes domain where generated code must satisfy strict language-level, security, and software-engineering constraints. Existing benchmarks and metrics remain insufficient for repository-level Solidity generation, where models must synthesize complete contracts from natural language requirements. To address this gap, we introduce SolidityBench, a benchmark of 5,470 repository-level Solidity smart contracts paired with natural language descriptions. We also propose SolidityScore, a Solidity-aware semantic metric that emphasizes domain-critical constructs such as security modifiers, contract declarations, and Solidity-specific keywords. Using this benchmark, we evaluate representative code LLMs, including Qwen2.5-Coder, DeepSeek-Coder, and CodeLlama, across zero-shot prompting, Chain-of-Thought reasoning, in-context learning, retrieval-augmented generation, and supervised fine-tuning. The results show that general-purpose models exhibit systematic structural deficiencies in repository-level Solidity generation. Among non-parametric methods, retrieval-augmented generation performs best, while in-context learning degrades beyond two examples due to context saturation. Supervised fine-tuning achieves the largest improvement by internalizing Solidity-specific constraints into model parameters. Overall, our study provides a comprehensive benchmark for repository-level Solidity code generation and shows that high-quality domain data combined with supervised fine-tuning is the most effective strategy for improving the reliability of LLM-generated smart contracts.

Problem

• Solidity
• code generation
• large language models
• smart contracts
• repository-level

Innovation

• SolidityBench
• SolidityScore
• repository-level code generation
• supervised fine-tuning
• retrieval-augmented generation