A Large Scale Empirical Analysis on the Adherence Gap between Standards and Tools in SBOM

2026-01-09

ACM Transactions on Software Engineering and Methodology

AI Summary

This study addresses the widespread non-compliance of Software Bill of Materials (SBOM) tools with established standards, which poses significant compliance risks and supply chain security vulnerabilities. We present the first large-scale, two-phase empirical investigation, introducing an automated evaluation framework named SAP to systematically assess six widely used SBOM generation tools. Our analysis encompasses 55,444 SBOMs produced from 3,287 real-world repositories, evaluating baseline performance and longitudinal changes over one year. The findings reveal substantial gaps in policy compliance, alarmingly low cross-tool consistency (as low as 7.84%), and poor license identification accuracy (under 20%). To support future research and tool improvement, we publicly release our complete evaluation framework and dataset, providing empirical foundations for advancing SBOM standardization and implementation.

Technology Category

• Constraint Satisfaction and Optimization: Solvers and Tools
• Search and Optimization: Evaluation and Analysis
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

A Software Bill of Materials (SBOM) is a machine-readable artifact that systematically organizes software information, enhancing supply chain transparency and security. To facilitate the exchange and utilization of SBOMs, organizations such as the Linux Foundation and OWASP have proposed SBOM standards. Following standards, organizations have developed tools for generating and utilizing SBOMs. However, limited research has examined the adherence of these SBOM tools to standard specifications, a gap that could lead to compliance failures and disruptions in SBOM utilization. This paper presents the first large-scale, two-stage empirical analysis of the adherence gap, using our automated evaluation framework, SAP. The evaluation, comprising a baseline evaluation and a one-year longitudinal follow-up, covers 55,444 SBOMs generated by six SBOM tools from 3,287 real-world repositories. Our analysis reveals persistent, fundamental limitations in current SBOM tools: (1) inadequate compliance support with policy requirements; (2) poor tool consistencies, including inter-tool consistency rates as low as 7.84% to 12.77% for package detection across languages, and significant longitudinal inconsistency, where tools show low consistency with their own prior versions; and (3) mediocre to poor accuracy for detailed software information, e.g., accuracy of package licenses below 20%. We analyze the root causes of these gaps and provide practical solutions. All the code, replication docker image, evaluation results are open sourced at GitHub and Zenodo1 for further researches.

Problem

• SBOM
• standards adherence
• tool compliance
• software supply chain
• empirical analysis

Innovation

• SBOM
• adherence gap
• empirical analysis
• tool consistency
• software supply chain security