MOS: Towards Effective Smart Contract Vulnerability Detection through Mixture-of-Experts Tuning of Large Language Models

2025-04-16

AI Summary

To address the low flexibility, poor interpretability, and high false-positive rate of existing smart contract vulnerability detection methods, this paper proposes an expert mixture-of-experts tuning (MoE-Tuning) framework based on large language models (LLMs). Our key contributions are: (1) a novel vulnerability-aware routing mechanism that dynamically activates domain-specific expert subnetworks according to semantic vulnerability patterns; and (2) a domain-enhanced MoE-Tuning paradigm integrating continual pretraining, LLM-generated + expert-verified data construction, parallel expert feedforward inference, and a dual-objective loss comprising detection performance and entropy-based expert specialization regularization. Experiments demonstrate improvements of +6.32% in F1-score and +4.80% in accuracy over strong baselines. Joint human–LLM evaluation yields 82.96%, 85.21%, and 94.58% scores (on a 4-point scale) for explanation correctness, completeness, and conciseness, respectively.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Smart contract vulnerabilities pose significant security risks to blockchain systems, potentially leading to severe financial losses. Existing methods face several limitations: (1) Program analysis-based approaches rely on predefined patterns, lacking flexibility for new vulnerability types; (2) Deep learning-based methods lack explanations; (3) Large language model-based approaches suffer from high false positives. We propose MOS, a smart contract vulnerability detection framework based on mixture-of-experts tuning (MOE-Tuning) of large language models. First, we conduct continual pre-training on a large-scale smart contract dataset to provide domain-enhanced initialization. Second, we construct a high-quality MOE-Tuning dataset through a multi-stage pipeline combining LLM generation and expert verification for reliable explanations. Third, we design a vulnerability-aware routing mechanism that activates the most relevant expert networks by analyzing code features and their matching degree with experts. Finally, we extend the feed-forward layers into multiple parallel expert networks, each specializing in specific vulnerability patterns. We employ a dual-objective loss function: one for optimizing detection and explanation performance, and another for ensuring reasonable distribution of vulnerability types to experts through entropy calculation. Experiments show that MOS significantly outperforms existing methods with average improvements of 6.32% in F1 score and 4.80% in accuracy. The vulnerability explanations achieve positive ratings (scores of 3-4 on a 4-point scale) of 82.96%, 85.21% and 94.58% for correctness, completeness, and conciseness through human and LLM evaluation.

Problem

• Detects smart contract vulnerabilities using expert-tuned large language models
• Reduces false positives and improves explanation quality in vulnerability detection
• Enhances detection accuracy and F1 score over existing methods

Innovation

• Mixture-of-experts tuning for LLM adaptation
• Vulnerability-aware routing for expert activation
• Dual-objective loss for performance optimization