How Do Solidity Versions Affect Vulnerability Detection Tools? An Empirical Study

2025-04-07

AI Summary

This study investigates how Solidity version evolution impacts the compatibility, effectiveness, and performance of smart contract vulnerability detection tools. We conduct a large-scale empirical evaluation across Solidity versions 0.4.x–0.8.x using the SmartBugs framework, integrating mainstream tools including Mythril and Slither on a corpus of real-world contracts. Our automated assessment systematically measures compilation compatibility, vulnerability detection rates, and execution time. We propose the first three-dimensional evaluation framework—“pragma compatibility–detection effectiveness–runtime performance”—to holistically characterize tool behavior under language evolution. Results reveal that over 60% of tools suffer compilation failures or increased false negatives on newer Solidity versions, with pragma constraints exhibiting a statistically significant, strong correlation with detection degradation. The findings provide empirically grounded guidance and a reproducible methodology for sustaining tool robustness amid language updates, thereby fostering co-evolution of vulnerability detection techniques and the Solidity language.

Technology Category

• Constraint Satisfaction and Optimization: Solvers and Tools
• Search and Optimization: Evaluation and Analysis
• Machine Learning: Evaluation and Analysis

Application Category

• Security and Privacy: Cryptocurrency and smart contracts
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Context: Smart contract vulnerabilities pose significant security risks for the Ethereum ecosystem, driving the development of automated tools for detection and mitigation. Smart contracts are written in Solidity, a programming language that is rapidly evolving to add features and improvements to enhance smart contract security. New versions of Solidity change the compilation process, potentially affecting how tools interpret and analyze smart contract code. Objective: In such a continuously evolving landscape, we aim to investigate the compatibility of detection tools with Solidity versions. More specifically, we present a plan to study detection tools by empirically assessing (i) their compatibility with the Solidity pragma directives, (ii) their detection effectiveness, and (iii) their execution time across different versions of Solidity. Method: We will conduct an exploratory study by running several tools and collecting a large number of real-world smart contracts to create a balanced dataset. We will track and analyze the tool execution through SmartBugs, a framework that facilitates the tool execution and allows the integration of new tools.

Problem

• Assess compatibility of detection tools with Solidity versions
• Evaluate effectiveness of tools in detecting vulnerabilities
• Measure execution time of tools across Solidity versions

Innovation

• Assessing compatibility with Solidity pragma directives
• Evaluating detection effectiveness across versions
• Measuring execution time using SmartBugs framework