A Large Scale Study of AI-based Binary Function Similarity Detection Techniques for Security Researchers and Practitioners

2025-11-02

AI Summary

Existing AI-driven binary function similarity detection (BFSD) tools suffer from three key limitations: unclear performance determinants, absence of real-world scenario validation, and reliance on small-scale or low-quality datasets. Method: This paper introduces two large-scale, high-quality benchmark datasets—BinAtlas (7 million functions) and BinAres (54 real-world IoT firmware vulnerabilities)—and conducts the first systematic, real-world evaluation of nine state-of-the-art BFSD tools in IoT contexts. Contribution/Results: The study uncovers critical deficiencies, including poor cross-architecture robustness and low result consistency across tools. To address these, we propose a complementary tool fusion strategy that improves vulnerability detection accuracy by 13.4%. Our work bridges the gap between lab-based BFSD evaluation and practical, large-scale deployment, advancing BFSD toward industrial applicability.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Constraint Satisfaction and Optimization: Solvers and Tools
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Binary Function Similarity Detection (BFSD) is a foundational technique in software security, underpinning a wide range of applications including vulnerability detection, malware analysis. Recent advances in AI-based BFSD tools have led to significant performance improvements. However, existing evaluations of these tools suffer from three key limitations: a lack of in-depth analysis of performance-influencing factors, an absence of realistic application analysis, and reliance on small-scale or low-quality datasets. In this paper, we present the first large-scale empirical study of AI-based BFSD tools to address these gaps. We construct two high-quality and diverse datasets: BinAtlas, comprising 12,453 binaries and over 7 million functions for capability evaluation; and BinAres, containing 12,291 binaries and 54 real-world 1-day vulnerabilities for evaluating vulnerability detection performance in practical IoT firmware settings. Using these datasets, we evaluate nine representative BFSD tools, analyze the challenges and limitations of existing BFSD tools, and investigate the consistency among BFSD tools. We also propose an actionable strategy for combining BFSD tools to enhance overall performance (an improvement of 13.4%). Our study not only advances the practical adoption of BFSD tools but also provides valuable resources and insights to guide future research in scalable and automated binary similarity detection.

Problem

• Evaluating AI-based binary function similarity detection tools comprehensively
• Addressing limitations in performance analysis and dataset quality
• Assessing practical vulnerability detection in real-world IoT firmware

Innovation

• Constructed two large-scale high-quality binary datasets
• Evaluated nine representative AI-based BFSD tools
• Proposed combination strategy improving performance by 13.4%