Bounded Exhaustive Random Program Generation for Testing Solidity Compilers and Analyzers

2025-03-26

AI Summary

Random program generation for testing Solidity compilers and analyzers suffers from low guidance and poor deep-bug coverage. Method: This paper proposes a two-phase generation paradigm—“template-based generation + bounded exhaustive instantiation”: first, constructing abstract syntax tree templates containing bug-sensitive placeholders based on error-prone language features; second, performing bounded exhaustive instantiation of placeholders under Z3-satisfiability constraints, integrated with lightweight symbolic execution to preserve semantic validity and balance coverage with bug-triggering efficiency. Contributions/Results: The approach discovers 23 previously unknown vulnerabilities in solc, solang, and Slither; achieves coverage of 4,582 control-flow edges and 14,737 lines of code missed by solc’s unit tests; and significantly outperforms state-of-the-art Solidity fuzzers in both vulnerability detection and structural coverage.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Random program generators often exhibit opportunism: they generate programs without a specific focus within the vast search space defined by the programming language. This opportunistic behavior hinders the effective generation of programs that trigger bugs in compilers and analyzers, even when such programs closely resemble those generated. To address this limitation, we propose bounded exhaustive random program generation, a novel method that focuses the search space of program generation with the aim of more quickly identifying bug-triggering programs. Our approach comprises two stages: 1) generating random program templates, which are incomplete test programs containing bug-related placeholders, and 2) conducting a bounded exhaustive enumeration of valid values for each placeholder within these templates. To ensure efficiency, we maintain a solvable constraint set during the template generation phase and then methodically explore all possible values of placeholders within these constraints during the exhaustive enumeration phase. We have implemented this approach for Solidity, a popular smart contract language for the Ethereum blockchain, in a tool named Erwin. Based on a recent study of Solidity compiler bugs, the placeholders used by Erwin relate to language features commonly associated with compiler bugs. Erwin has successfully identified 23 previously unknown bugs across two Solidity compilers, solc and solang, and one Solidity static analyzer, slither. Evaluation results demonstrate that Erwin outperforms state-of-the-art Solidity fuzzers in bug detection and complements developer-written test suites by covering 4,582 edges and 14,737 lines of the solc compiler that were missed by solc unit tests.

Problem

• Focuses search space to find bug-triggering Solidity programs efficiently
• Generates random templates with bug-related placeholders for exhaustive testing
• Improves bug detection in Solidity compilers and analyzers compared to fuzzers

Innovation

• Bounded exhaustive random program generation
• Random program templates with placeholders
• Constraint-based exhaustive enumeration of values