Cross-Corpus Evaluation of Generalizable Vulnerability Detection in IoT Firmware

2026-08-11

AI Summary

This study addresses the challenges of vulnerability detection in IoT firmware—namely, ecosystem heterogeneity, resource constraints, and the absence of realistically labeled benchmarks—which hinder reliable evaluation of cross-corpus generalization. To this end, the authors introduce IoTVulBench, the first expert-reviewed benchmark with controlled contamination, and conduct a systematic evaluation of five model architectures, two fine-tuning approaches, and three curriculum learning strategies. Experimental results demonstrate that domain-matched data and thoughtfully designed curricula yield greater improvements in generalization than merely scaling up model size. The best-performing model achieves a Matthews Correlation Coefficient (MCC) of 0.73, outperforming the strongest static analysis tool by 0.42; it misses only 21% of vulnerabilities at a 0.5% false positive rate and retains 86% of its performance under identifier renaming attacks.

Abstract

IoT firmware vulnerability detection remains challenging due to heterogeneous firmware ecosystems, resource-constrained platforms, and limitations in existing benchmarks. Many datasets are synthetic or general-purpose and lack human-verified, contamination-screened annotations, limiting evidence on cross-corpus generalization across training sources, model architectures, and curriculum strategies. To address this gap, this paper introduces IoTVulBench, a human-verified benchmark for cross-corpus firmware vulnerability detection. IoTVulBench-Core was constructed from GitHub repositories, validated by three expert reviewers, and evaluated on a contamination-screened held-out target across five model architectures, two tuning methods, and three curriculum strategies, with ensemble, distillation, and robustness analyses. Models trained on IoTVulBench achieved the highest MCC among matched single-source datasets, reaching 0.58 versus 0.44 for PrimeVul and 0.39 for D2A. Staged curriculum learning increased MCC to 0.69, while a diversity-optimized ensemble achieved 0.73, improving by 0.42 MCC over the strongest reference comparator, a static analyzer at 0.31, and by 0.29 over PrimeVul. At a 0.5% false-positive rate, the model missed only 21% of vulnerabilities, compared with 71% for the strongest comparator. It retained 86% of its performance under identifier renaming and demonstrated strong calibration and largely faithful explanations. These findings indicate that domain-matched training data and curriculum design, rather than model scale alone, are key drivers of generalization in firmware vulnerability detection. The results provide a benchmark for future research and deployment-ready configurations for practical IoT security applications.

Problem

• IoT firmware
• vulnerability detection
• cross-corpus generalization
• benchmark dataset
• model generalization

Innovation

• cross-corpus generalization
• IoTVulBench
• curriculum learning
• firmware vulnerability detection
• human-verified benchmark