Understanding Binary Code Similarity for Real-World Vulnerability Detection: A Large-Scale Empirical Study

2026-06-27

AI Summary

Existing binary code similarity detection methods lack large-scale validation in real-world firmware scenarios, limiting their effectiveness in vulnerability discovery. This work systematically evaluates the impact of vulnerable function versions, vulnerability search space, function size, and compiler toolchains on detection performance using a dataset of 60,000 firmware images from 200 vendors. The study proposes two key optimizations: a build-aware query strategy and a third-party library (TPL)-aware two-stage search pipeline. Experimental results demonstrate that the build-aware query strategy improves Mean Reciprocal Rank (MRR) from 0.818 to 0.981, and the TPL-aware two-stage approach yields an additional 18.5% gain, substantially enhancing vulnerability detection capability in realistic settings.

Abstract

Firmware lies at the heart of IoT devices. Its development depends heavily on third-party libraries (TPLs), which greatly accelerate the process but simultaneously introduce associated vulnerabilities. Binary Code Similarity Detection (BCSD) is an effective technique for identifying vulnerabilities in firmware by comparing pairs of code segments. However, existing studies either evaluate their performance only on small-scale datasets or lack diversity in terms of vulnerabilities, TPLs, and firmware. Consequently, a comprehensive understanding of BCSD for real-world vulnerability detection remains absent. To bridge this gap, we conduct a large-scale study of vulnerability detection across 60,000 firmware images from 200 vendors using BCSD. Rather than introducing a novel model, we examine the influence of four key factors -- vulnerable function versions, vulnerability search space, function sizes, and compilation toolchains on BCSD performance. Our results reveal that these factors substantially affect performance, often by wide margins. To address this, we propose a build-aware query strategy that derives queries from representative real-world binaries, effectively closing the gap and raising the mean reciprocal rank (MRR) from 0.818 to 0.981. Furthermore, we demonstrate that a TPL-aware, two-stage search process significantly enhances accuracy, improving MRR by 18.5\% by limiting the search space.

Problem

• Binary Code Similarity Detection
• Firmware Vulnerability
• Third-Party Libraries
• IoT Security
• Empirical Study

Innovation

• Binary Code Similarity Detection
• Firmware Vulnerability Detection
• Build-Aware Query
• TPL-Aware Search
• Large-Scale Empirical Study